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About Research for Action 


Research for Action (RFA) is a Philadelphia-based nonprofit organization. We seek to use research as 
the basis for the improvement of educational opportunities and outcomes for traditionally underserved 
students. Our work is designed to strengthen public schools and postsecondary institutions; to provide 
research-based recommendations to policymakers, practitioners, and the public at the local, state, 

and national levels; and to enrich the civic and community dialogue about public education. For more 
information, please visit our website at www.researchforaction.org. 
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Executive Summary 


Introduction 


Children who have not learned to read by fourth grade are four times more likely to drop out of school, 
and this risk is even greater for low-income children.! In 2019, only a third of third-grade children at the 
School District of Philadelphia reached this important milestone, as measured by the Pennsylvania State 
Standardized Assessment.’ Early language and literacy skill delays are largely due to family economic 
disadvantages that, to fully address, would require broad-based policies to reduce poverty.’ Still, access to 
high-quality pre-K is increasingly recognized as a potential lever for preparing children disadvantaged by 
poverty to achieve developmentally-appropriate reading milestones as they age.* 


Children’s Literacy Initiative (CLI) aims to support the quality of pre-K environments and prepare 
children disadvantaged by poverty through its Blueprint for Early Literacy pre-K curriculum supplement 
teamed with workshops and individual coaching for pre-K educators. Table ES1 presents CLI’s theory of 
action for this work. 


Table ES1. Children’s Literacy Initiative’s support components and theory of action 


IMPLEMENTATION OUTCOMES FOR TEACHERS OUTCOMES FOR CHILDREN 


e Implementation of Blueprint e Increased teacher knowledge e Increased engagement in 

for Early Literacy pre-K and ability to implement language and literacy learning 
curriculum Supplement effective early language and 

literacy instruction e Increased mastery of early 
Professional development for literacy concepts and early 
pre-K educators through Increased teacher ability to language and literacy skills 
training and coaching create a positive classroom 

culture and a literacy-rich 

classroom environment 


About this Study 


Research for Action (RFA) conducted a three-year external evaluation of this program from 2017-2019. 
In this study, we sought to understand CLI’s impact on teachers’ knowledge and ability to implement 
best practices for instructing language and literacy, classroom language and literacy environments, and 
children’s engagement and skills in language and literacy. To do this, we conducted a rigorous causal 
impact study that employed a mixed-methods quasi-experimental research design involving 11 centers 
receiving CLI professional development and the Blueprint for Early Literacy curriculum supplement 
and 11 statistically similar centers serving as a comparison group. We also assessed the professional 
development CLI provides through training workshops and individual content-focused coaching for pre-K 
educators. This study draws on cross-verification of multiple methods of data collection and analysis: 
direct assessments of children’s vocabulary, researcher observations of the classroom language and 
literacy environments, end-of-year teacher surveys, and interviews with CLI staff, center directors, and 
teachers. 


1 Hernandez, Donald J (2011). “Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation.’ Retrieved from https://files.eric.ed.gov/fulltext/ 
£D518818.pdf 

2 School District of Philadelphia. (2020, February 10). SY 2018-19 District Scorecard. Retrieved January 2019 from https://www.philasd.org/performance/programsservices/ 
school-progress-reports/district-scorecard/ 

3 Duncan, G. J. & Magnuson, K. A. (2005). Can family socioeconomic resources account for racial and ethnic test score gaps? The Future of Children, 15, 35-54. 

4 Magnuson, K. A., Ruhm, C. J., & Waldfogel, J. (2004). Does prekindergarten improve school preparation and performance? (NBER Working Paper 10452). Cambridge, MA: National 
Bureau of Economic Research: https://www.nber.org/papers/w10452.pdf 
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Summary of Findings 


Positive impacts despite higher-than-typical teacher turnover, along with teacher and 
center director descriptions of ease of use, suggest that CLI's early language and literacy 
supports are particularly well-suited for improving quality and outcomes in high-need 
pre-K environments. 


In pre-K centers receiving CLI supports, teachers were more able to implement effective instructional 
practices and created richer classroom literacy environments, and children’s vocabulary grew faster by an 
order of magnitude of two-and-a-half months, relative to teachers and children in similar pre-K centers. 
We observed these impacts consistently throughout the study even though year-to year teacher turnover 
was higher than national averages. We found that challenges associated with the high turnover inhibited 
consistently robust implementation of the CLI’s intervention. Fewer than one in five teachers received the 
full intended level of support, and less than half of surveyed teachers reported implementing with fidelity 
all three of the key instructional approaches of the Blueprint for Early Literacy curriculum supplement 
(Message Time Plus, Intentional Read Aloud, and Power of Three). Nevertheless, despite variable dosage, 
the study finds statistically significant positive impacts for student learning and teacher practice. 
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Children who have not learned to read by fourth grade are four times more likely to drop out of school, 
and this risk is even greater for low-income children.‘ In 2019, only a third of third-grade children at the 
School District of Philadelphia reached this important milestone, as measured by the Pennsylvania State 
Standardized Assessment.’ Early language and literacy skill delays are largely due to family economic 
disadvantages that, to fully address, would require broad-based policies to reduce poverty.’ Still, access to 
high-quality pre-K is increasingly recognized as a potential lever for preparing children disadvantaged by 
poverty to achieve developmentally-appropriate reading milestones as they age.* 


There is a gap between “research-based” and “evidence-based” curricula to improve early 
language and literacy skills 


With the goal of supporting a high-quality learning environment, most publicly-funded preschool programs 
require the use of a “research-based” curriculum.® Pennsylvania requires demonstration of alignment of 
pre-K curricula to early learning standards formulated with guidance from practitioners and program 
specialists.° Yet several popular early childhood curricula that meet state requirements offer limited or no 
empirical evidence of impact on child outcomes, including Creative Curriculum.”®° Creative Curriculum 

is utilized widely both nationally and in Philadelphia; three-fourths of Head Start programs nationally 
reported using Creative Curriculum in 2017,”° and it has been adopted in the School District of Philadelphia 
and in the city’s universal pre-K program (PHL pre-k). Local stakeholders indicate that they supported 
wide adoption of Creative Curriculum because it is aligns with state early learning standards and is 
therefore on the state’s approved list of Pre-K curricula; it offers the benefit of an aligned assessment; and 
their teachers report that it is easy to use. Broad adoption of a single curriculum also facilitates the delivery 
of consistent professional development and support. 


Children’s Literacy Initiative developed Blueprint for Early Literacy to fill this gap and 
support children disadvantaged by poverty 


Children’s Literacy Initiative aims to address the challenge of supporting pre-K programs serving high- 
needs families and fill the gap in evidence-based curricula. CLI is a Philadelphia-based non-profit 
working with pre-K through 5th grade teachers to improve early literacy instruction and close the literacy 


1 Hernandez, Donald J (2011). “Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation.’ Retrieved from https://files.eric.ed.gov/fulltext/ 
ED518818.pdf 

2 School District of Philadelphia. (2020, February 10). $Y 2078-19 District Scorecard, Retrieved January 2019 from https://www.philasd.org/performance/programsservices/ 
school-progress-reports/district-scorecard/ 

3 Duncan, G. J. & Magnuson, K. A. (2005). Can family socioeconomic resources account for racial and ethnic test score gaps? The Future of Children, 15, 35-54. 

4 Magnuson, K. A., Ruhm, C.J., & Waldfogel, J. (2004). Does prekindergarten improve school preparation and performance? (NBER Working Paper 10452). Cambridge, MA: National 
Bureau of Economic Research: https://www.nber.org/papers/w10452. pdf 

5 Jenkins, J. M., Whitaker, A. A., Nguyen, T., & Yu, W. (2019). Distinctions without a difference? Preschool curricula and children’s development. Journal of Research on Educational 
Fffectiveness, 12, 514-549. 

6 Pennsylvania Department of Education (2020). Learning Standards for Early Childhood — Infant/Toddler; Pre-Kindergarten; Kindergarten; Grade 7 and Grade 2. Retrieved from 
https://www.education.pa.gov/Early%20Learning/Early%20Learning%20Standards/Pages/Infant-Toddler-Pre-K-Learning-Standards.aspx 

7 Preschool Curriculum Evaluation Research Consortium (2008). Effects of Preschool Curriculum Programs on School Readiness (NCER 2008-2009). Washington, DC: National Center for 
Education Research, Institute of Education Sciences, U.S. Department of Education. Washington, DC: U.S. Government Printing Office. 

8 National Center on Early Childhood Development, Teaching, and Learning. (2019). Curriculum Consumer Report: Preschool. Retrieved from https://eclkc.ohs.acf.hhs.gov/sites/de- 
fault/files/featured_file/preschool-curriculum-consumer-report-032519.pdf 

9 Institute of Education Sciences. (2013). What Works Clearinghouse Intervention Report: 7he Creative Curriculum for Preschool, Fourth Edition. Retrieved from: https://ies.ed.gov/ 
ncee/wwe/Docs/InterventionReports/wwc_creativecurriculum_030513.pdf 

10 Bernstein et al. (2018). A Portrait of Head Start Classrooms and Programs: FACES Spring 2017 Data Tables and Study Design. OPRE Report 2019-10. Washington, DC: Office of 
Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. 
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achievement gap between disadvantaged children and their more affluent peers. CLI’s supports for pre-K 
language and literacy combine an in-person professional development approach to teacher training and 
support for implementing instructional approaches with its Blueprint for Early literacy curriculum, which 
can be used on its own or supplement existing classroom curricula.'? 


Research for Action conducted a three-year study of implementation and outcomes to 
assess the quality and impact of CLI's supports for pre-K language and literacy 


This report examines implementation and outcomes of CLI’s supports as implemented in 11 high-quality 
Philadelphia pre-K centers over a two-and-a-half-year period, from January 2017 through June 2019. 
The study follows the 35 classrooms in these centers to track implementation successes and challenges 
as teachers implemented Blueprint alongside Creative Curriculum. This study also compares outcomes 
in these centers to those in 11 similar centers to assess CLI’s impact on children, teachers, and classroom 
environments. 


Wan @] ce F-1aly4-1 t(e)a mein dal-mn\-) olelat 


This report is comprised of five sections that describe CLI’s supports for pre-K language and literacy; 
provide evidence of impact for children, teachers, and classroom environments; and present an 
examination of the extent and quality of implementation of CLI’s professional development and Blueprint 
for Early Literacy. 


Section 1: Guiding Research Questions and Study Design. In this section, we provide an 
overview of the research questions, study design, characteristics of the study population, and data 
collection methods. This section also describes the focus of the study during each study year. 


Section 2: A Description of CLI’s Supports for Pre-K Language and Literacy. In this section, we 
draw on observations of trainings, a review of curricular and training materials, and interviews 
with CLI staff and coaches to provide a rich description of CLI’s Blueprint for Early Literacy 
curricular materials, professional development, and instructional approaches. 


Section 3: Impact of CLI’s Supports for Pre-K Language and Literacy. Drawing on CLI’s theory 
of action to guide our observations of impact, we examined outcomes for children, teachers, and 
classrooms. This mixed-methods causal impact study compares outcomes in 11 CLI-supported 
centers to a statistically equivalent group of 11 centers not receiving CLI supports. Data are 
triangulated across multiple methods to present a coherent, multi-faceted evaluation of impact. 


Section 4: Implementation of CLI’s Supports in Philadelphia. This section presents results of a 
descriptive assessment of the extent and quality of CLI’s teacher professional development and 
Blueprint curriculum implementation. We explore the successes and challenges of implementation 
in Philadelphia, a setting characterized by high need and above-average teacher turnover rates. 
These analyses draw on multiple methods of data collection, including administrative data on 
training and coaching, surveys of teachers, interviews with CLI staff and coaches, and interviews 
with center directors and teachers. 


Section 5: Conclusions and Implications. Our report ends with a review of key findings and 


implications for CLI’s professional development approach and curriculum and a discussion of 
broader implications for early childhood decision-makers and practitioners. 


11 https://cliblueprint.org/ 
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Section 1: Guiding Research Questions and Study Design 


Research for Action (RFA) conducted a three-year study of impact and implementation of CLI’s Blueprint 
for Early Literacy. This section describes the research questions and study design that guided our 
approach to evaluation. We also discuss characteristics of the study population and describe our data 
collection activities. More information on study design and data collection can be found in the Appendix. 


A. Research Questions 
The research questions that guided this study are as follows: 


1. What is the growth in the engagement and early language and literacy skills of children in 
classrooms with CLI Blueprint materials, training, and coaching, compared to children in similar 
classrooms without CLI supports? 


2. What is the growth in teacher knowledge and ability to implement instructional practices in pre-K 
centers with the CLI supports, compared to teachers in similar classrooms without CLI supports? 


3. What is the impact of CLI Blueprint on the early language and literacy environment in classrooms 
in pre-K centers with CLI supports, compared to classrooms without CLI supports? 


4. What is the extent and quality of implementation of CLI’s professional development and Blueprint 
for Early Literacy? 


B. Study Design 


This section describes the design of this research study, including the phases of the research, a description 
of data collection activities, study site selection, and characteristics of the study population. 


I. Phases of research 


Our research was phased over three years to understand both the implementation and impact. 


e Year One (Spring 2017). The primary goal of the first year of research was to understand 
the research base for Blueprint for Early Literacy, factors that support strong implementation; 
training, and coaching; and cross-cutting challenges. While we refer to this phase as “Year One,” 
the research began in January 2017 and represents six months of data collection. 


Year Two (SY 2017-18). In Year Two, we examined several specific challenges identified in 
Year One: providing professional development in the context of high teacher turnover; efforts to 
integrate Blueprint with curricula already in use; and differentiating instruction for all learners. 
We also began assessing the impact of CLI’s supports on children, teachers, and classroom 
environments. 


Year Three (SY 2018-19). In Year Three, we continued to evaluate impact, and our 


implementation research focused more deeply on coaching as well as on the role of assistant 
teachers in Blueprint implementation. 


Figure 1 provides a graphic illustration of this phasing. 
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Figure 1. Focal Areas of RFA’s Evaluation of CLI’s Supports for Pre-K Language and Literacy by Study Phase, 


2017-2019 


Understanding 
Blueprint, factors that 
support implementation, 
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impact of Blueprint 
on children’s literacy 
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practice; 


RESEARCH for ACTION 
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instruction for 
all learners. 


Il. Data Collection Activities 


This study included the following data collection activities (see Appendix for more information): 
All Three Years: 


CLI administrative data on training and coaching hours for lead teachers. CLI provided data 
on participation in training and coaching for lead teachers each year of the study.” 


Interviews with CLI staff and coaches. In the spring of each study year, RFA interviewed relevant 
CLI staff and all four CLI coaches to deepen understanding of CLI’s support, how Blueprint works, 
and of CLI’s expectations for teachers implementing Blueprint. 


Interviews with a sample of center directors and teachers at centers receiving CLI supports. 
For a better understanding of perceptions of CLI’s professional development and curricular 
materials, RFA interviewed a sample of center directors and teachers purposively selected each 
study year (N=69 total interviews). The Appendix describes sample selection criteria for each 
study year. 


Observations of CLI professional development meetings and trainings. Throughout the 
study period, RFA conducted 10 observations of CLI supports: four director meetings, one teacher 
meeting, and five CLI trainings. These observations provided background information to support 
the analysis of CLI professional development and Blueprint implementation. 


Document review. RFA reviewed curriculum and training materials to support the analysis of CLI 
professional development and Blueprint implementation. 


Years Two and Three only: 


e Beginning- and end-of-year direct assessments of children’s early language and literacy 
skills, using the Peabody Picture Vocabulary Test, a measure of receptive vocabulary (N = 808 in 
Year Two and 783 in Year Three).?° 


End-of-year observations of classroom language and literacy environments using the Early 
Language and Literacy Classroom Observation Pre-K tool** (N=74 classrooms each year). 


Year Three only: 


e End-of-year teacher survey administered online in Spring 2019 to lead pre-K teachers to assess 
teacher perceptions of children’s outcomes and teacher knowledge and ability to implement 
effective instructional practices in early language and literacy (response rate rate=74%). Lead 
teachers in Blueprint centers were also surveyed about their perceptions and experiences with 
CLI training, coaching, and Blueprint for Early Literacy (response rate = 75%). We also surveyed 
assistant teachers about their role in whole group instruction (response rate = 65%). 


Ill. Study site selection and characteristics of study population 


This study was set in 22 Philadelphia pre-K centers recruited by CLI and selected for participation by 
Research for Action. The sample of centers includes 4 school-based centers and 18 community-based 
providers. Forty-two percent of classrooms were Head Start classrooms. Five were PHL pre-k classrooms. 


12 We define “lead” teacher as the CLI teacher targeted to receive coaching. In some cases, this teacher was not officially a lead teacher but an assistant acting as lead because of 
staffing challenges. In other cases, there were technically two lead teachers in the classroom. However, CLI typically only coached one “lead” teacher per class. 

13 Dunn, L. M., Dunn, D. M., Lenhard, A., Lenhard, W., & Suggate, S. (2015). Peabody Picture Vocabulary Test [manual]. Pearson. 

14 Smith, M. W., Brady, J. P.,, & Anastasopoulos, L. (2008). User's Guide to the Early Language & Literacy Classroom Observation Pre-K Tool. Baltimore: Paul H. Brookes Publishing Co. 
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CLI used the following specific criteria to select centers for participation in this study: 


Centers that met a minimum quality standard (i.e., at least a STAR 3 rating in Pennsylvania’s 
Keystone STARS system?) in 2017; 


Centers with strong director buy-in and a willingness to adopt a new or additional curriculum; 
and 


Centers located in low-income areas with a high need for quality childcare. According to recent 
census data, study centers were in neighborhoods where at least a third of families were living in 
poverty.'® 


To ensure that we could assess impact and interpret outcome differences in Blueprint and comparison 
sites as causal, we selected comparison centers that were statistically similar, on average, with respect 

to characteristics of classrooms (e.g., size) and children (e.g., pretest vocabulary skills) related to early 
literacy instruction and outcomes. Our approach was to recruit four more comparison sites than needed, 
compare sites to those receiving CLI supports on key factors, and choose a subsample of 11 comparison 
centers similar on average to Blueprint centers (see Appendix for details). Table 1 provides some 
descriptive information about the classrooms and children in Blueprint and comparison, or “business-as- 
usual,” classrooms. 


Table 1. Characteristics of study classrooms and children, 2017-2019 
BLUEPRINT BUSINESS-AS-USUAL 
CHARACTERISTICS OVERALL CLASSROOMS CLASSROOMS 
Curriculum being implemented Creative Curriculum and Blueprint Creative Curriculum 
Number of pre-K centers 22 11 dal 
Number of pre-K classrooms? 74 36 38 


% of Head Start classrooms 


Average number of pre-K classrooms in 
Ponieie SS) 3.4 3.7 


Average class size 14.5 14.7 14.2 
Number of children in study aL axel 727 864 
Study Year Two (2017-18) 808 372 436 
Study Year Three (2018-19) 783 355 428 
Age at start of school year 4.3 years 4.2 years 4.3 years 
% Identified by teacher as EL° 8.0% 6.9% 8.9% 
Pre-test vocabulary assessment score® 110.97 111.69 110.36 


‘The number of open classrooms enrolled in our study changed from year-to-year, with some classrooms newly opened and others closed during the study period. We report 
classroom characteristics for classrooms in 2018-19 in this table. See Appendix for more details. 


Children identified by their teachers as English Learners were pre-screened for English proficiency prior to assessment. Children who did not pass the screener were not assessed. 
Thus, the child assessment analytic sample described here underestimates the number of EL children in study classrooms. 


‘Fall (pre-test) Peabody Picture Vocabulary Test growth scale score 


15 The Pennsylvania Key (2018). Keystone Stars. Retrieved from https://www.pakeys.org/keystone-stars/. 
16 United States Census Bureau. (2017). TIGER/Line Shapefiles Data. Retrieved from https://www.census.gov/geo/maps-data/data/tiger-line.html. 
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Curriculum being implemented in study sites. \n this study, Blueprint for Early Literacy was 
implemented alongside Creative Curriculum in centers receiving CLI supports, except for one center, 
which was implementing Acelero’s Ready to Shine curriculum. Similarly, all but one comparison center 
was implementing Creative Curriculum; the remaining center implemented HighScope. Thus, the 
treatment contrast or comparison for the impact study is Blueprint for Early Literacy plus Creative 
Curriculum compared to Creative Curriculum alone. 


Characteristics of classrooms and children. Overall, there were 74 classrooms in the study, nearly half 
(42%) of which were Head Start classrooms. On average, there were between 3 and 4 classrooms in 

each center, with each classroom enrolling between 14 and 15 children. Additionally, this study involved 
vocabulary assessments of more than 1,600 children during the last two years of the study—about 800 
children each year. The children in our assessment sample were, on average, a little younger than four- 
and-a-half years old when they started pre-K, and about 8% of children were identified by their teachers 
as English Learners.”’ Importantly, the children in CLI-supported and business-as-usual classrooms 
scored similarly on the pre-test assessment of vocabulary, strengthening our assumption that differences 
in post-test assessments can be attributed to CLI supports. 


17 Because our primary outcome for children was English vocabulary, we only included English Learners who passed an English proficiency screener. 
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Section 8: A Description of CLI Supports for Pre-K 
Language and Literacy 


There are two components of CLI’s supports for pre-K language and literacy: 1) Curricular materials 
paired with research-based instructional practices and 2) CLI’s professional development for classroom 
teachers. To provide a rich description of these components, we analyzed data from observations of 
trainings, curricular and training materials, and interviews with CLI staff, and coaches. 


|. Support Component #1: Blueprint for Early Literacy curriculum supplement 
and research-based instructional practices 


The Blueprint for Early Literacy curriculum supplement includes pre-K literacy materials such as daily 
lesson plans, more than 100 children’s books, and 10 structured and sequential theme guides. Blueprint 
uses best practices in early childhood programs to strengthen language and literacy skills through play- 
based instruction.'® The implementation of the curriculum, which can either be used on its own or paired 
with a curriculum already in use, is structured around three key elements: 


e Message Time Plus (MTP) is a daily modeled writing and shared reading instructional practice. 
Teachers brainstorm, plan, and write text in front of children, then engage in a shared reading of 
the text. 


Intentional Read Aloud (IRA) is a daily structured practice during which teachers model fluent 
reading and reading behaviors as teachers and children think about, talk about, and respond to 
text before, during, and after reading. 


Power of Three is a tool to promote a classroom culture of responsibility and engagement. Power 
of Three is designed to shift teachers away from focusing on classroom rules toward encouraging 
children and teachers to share classroom responsibilities. Using the Power of Three as a 
framework integrated into the culture of the classroom, teachers seek to help children gain skills 
to “Take care of yourself, each other, and our classroom.” 


18 NAEYC (2018). NAFYC Early Learning Program Accreditation Standards and Assessment Items. Retrieved from https://www.naeyc.org/sites/default/files/globally-shared/down- 
loads/PDFs/accreditation/early-learning/standards_and_assessment_web_0.pdf 
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Il. Support Component #2: CLI Professional development for classroom teachers 


CLI offers professional development for pre-K teachers through workshops on CLI’s practices for early 
literacy instruction and individual coaching for educators. 


CLI trainings in instructional approaches. All lead and assistant pre-K teachers were invited to 
participate in three core half-day Blueprint trainings: 1) Introduction to Blueprint, 2) Message 
Time Plus (MTP), and 3) Intentional Read Aloud (IRA). These trainings, which were offered 
throughout the school year, were designed to introduce teachers to key concepts in early language 
and literacy instruction. The presentation of concepts was grounded in the research evidence 
upon which Blueprint was founded.” The goal of trainings was to train teachers in CLI’s pre-K 
instructional approaches—MTP, IRA, and Power of Three. 


CLI content-focused coaching. Additionally, CLI paired each lead teacher with a coach that 
provides 20 hours of in-person, content-focused coaching annually. The goal of Blueprint coaching 
was for teachers to develop a solid understanding of Blueprint and its practices, increase teacher 
confidence and comfort with the curriculum, and strengthen teachers’ adaptations of Blueprint 

to meet the needs of their learners. Coaching content was tailored to teachers’ needs and evolved 
as teachers become more experienced with Blueprint. Each session in the CLI content-focused 
coaching model included three key elements: 1) a pre-conference, 2) lesson implementation, 

and 3) a post-conference. Typically, during the pre-conference, coaches and teachers focused 

on the objectives of the lesson to follow. Then, either the teacher, the coach, or both together 
implemented the lesson. In the post-conference the coach and teacher reflected on the lesson and 
identified what to work on between coaching sessions. This time for feedback and reflection is one 
of the central elements of an effective professional learning experience for teachers.”° 


19 £.g., Strickland, D. S. & Riley-Ayers, S. (2006). Early Literacy: Policy and Practice in the Preschool Years. (National Institute for Early Education Research) Preschool Policy Brief Issue 
10 Retrieved from http://nieer.org/policy-issue/policy-brief-early-literacy-policy-and-practice-in-the-preschool-years & 

Yoshikawa et al. (2013). Investing in Our Future: The Evidence Base on Preschool Education. Retrieved from Society for Research in Child Development: https://www.fcd-us.org/ 
assets/2013/10/Evidence20Base200n20Preschool20Education20FINAL.pdf 

20 Darling-Hammond, L., Hyler, M. E., & Gardner, M. (2017). Effective Teacher Professional Development. Palo Alto, CA: Learning Policy Institute. 
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Section 3: Impact of CLI’s Pre-K Language and Literacy 
ob he) oleyarsmoyam @labilohw-valuM h-y-Kelak-vucme-W ale Mm OJE-h-1<3 weleyeats) 


Together, the core components of CLI’s pre-K language and literacy supports-Blueprint for Early Literacy 
and CLI training and coaching for teachers—are expected to improve outcomes for children and teachers 
and improve classroom language and literacy environments. Table 2 outlines CLI’s supports and theory of 
action, which provided a framework for our impact evaluation. 


Table 2. Children’s Literacy Initiative’s Pre-K Language and Literacy Supports 


IMPLEMENTATION OUTCOMES FOR TEACHERS OUTCOMES FOR CHILDREN 


e Implementation of Blueprint e Increased teacher knowledge e Increased engagement in 

for Early Literacy pre-K and ability to implement language and literacy learning 
curriculum supplement effective early language and 

literacy instruction e Increased mastery of early 
Professional development for literacy concepts and early 
pre-K educators through Increased teacher ability to language and literacy skills 
training and coaching create a positive classroom 

culture and a literacy-rich 

classroom environment 


To assess impact on outcomes, we conducted mixed-methods quasi-experimental study combining data 
from a range of sources: direct assessments of children’s vocabulary skills, observations of classroom 
environments, surveys and interviews with lead teachers, and interviews with center directors. 


This section of the report is divided into three sub-sections: 


A. Impact on Children’s Skills and Engagement in Early Language and Literacy Activities. 
This sub-section examines the impact of Blueprint on pre-K children’s vocabulary gains. This 
section also examines teacher and center director perceptions of how Blueprint affects student 
engagement and learning. 


Impact on Teacher Knowledge and Practice. This sub-section examines impact of CLI’s pre-K 
language and literacy supports on teacher perceptions of knowledge and ability to implement 
effective language and literacy instructional practices. We also present findings on teacher 
self-reported frequency of implementing best instructional practices for teaching reading, 

oral language, and writing skills; and we asked Blueprint teachers to assess the utility of the 
curriculum for supporting specific teacher instructional practices such as planning a variety of 
activities and management classroom behavior. . 


Impact on Classroom Language and Literacy Environments. In this sub-section, we examine 
impact of CLI’s pre-K language and literacy supports on multiple dimensions of classroom 
language and literacy environments. 
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CLI expects that by helping teachers increase their knowledge and skills in early language and literacy 
instruction, children will in turn become more engaged in language and literacy learning and gain 
language and literacy skills more rapidly than children otherwise would. In this section, we report 
evidence of the impact of CLI’s pre-K language and literacy supports on children’s early language and 
literacy skills. We report our findings from direct assessments of children’s oral language skills as well as 
teacher perceptions of impact on skills and engagement in classroom activities. 


|. Direct Assessments of Children’s Oral Language Skills 


To assess impacts of CLI’s pre-K language and literacy supports on children’s early language and literacy 
skills, we triangulated evidence from direct assessments of children’s oral language skills in SY 2017-18 
and 2018-19 (described below), interviews with center directors and lead teachers in Spring 2017 and 
2018, and surveys of lead teachers in Spring 2019. 


Key Findings: Impact on Children’s Outcomes 


e Direct assessments of children’s receptive vocabulary showed that children in Blueprint 
classrooms outpaced their peers by two-and-a-half months of additional growth from fall 
to spring. 


e Teachers who used Blueprint materials and approaches to instruction reported that their 
children learned a variety of foundational and higher-level skills for reading and writing. 


e Teachers also described how the routines and intentionality of Blueprint supported 
children’s engagement, including the specific elements of Message Time Plus and 
Intentional Read Aloud. 


To assess impact on oral language skills, we observed growth over time in children’s receptive vocabulary. 
Receptive vocabulary refers to the words that someone can comprehend, a key feature of early language 
acquisition and is strongly correlated with later literacy.21 To measure receptive vocabulary, we employed 
the Peabody Picture Vocabulary Test (PPVT), a 10-minute assessment administered by trained assessors 
to each child twice per year in Years Two and Three of the study.” 


Children in Blueprint classrooms outpaced their peers in vocabulary development by 
two-and-a-half months of growth 


The first cohort of children attended pre-K in 2017-18 (N=808) and the second in 2018-19 (N=783). We 
assessed their baseline receptive vocabulary skills in the fall, before receiving instruction, and again in 
the spring. Then, we compared growth from fall to spring in Blueprint classrooms to classrooms where, 
on average, children scored similarly in the fall. The table that follows presents the results from our 
analysis.”° 


We present results disaggregated by study year and combined across study years (see Appendix for more 
information on our analysis). 


21 Lonigan, C. J., and Shanahan, T. (2009) “Developing Early Literacy: Report of the National Early Literacy Panel. Executive Summary. A Scientific Synthesis of Early Literacy 
Development and Implications for Intervention.” Retrieved from the National Institute for Literacy: https://lincs.ed.gov/publications/pdf/NELPSummary.pdf. 

22 Dunn, L. M., Dunn, D. M., Lenhard, A., Lenhard, W., & Suggate, S. (2015). Peabody Picture Vocabulary Test [manual]. Pearson. 

23 See Appendix for full statistical models, year-specific results, and a description of our approach to pooling data across years. 
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Table 3. Estimated spring children’s vocabulary scores in Blueprint and business-as-usual classrooms, 2017-2019 


Scores for... Classrooms Classrooms (ORE) additional growth 
All children (n=1,591) 119.7 116.7 3.0 (0.001) DAS 
Year Two (n=808) 120.0 116.4 3.6 (0.005) 2.8 
Year Three (n=783) 119.6 116.9 2.7 (0.021) 2.4 


Source: RFA assessements of children’s receptive vocabulary using the Peabody Picture Vocabulary Test (PPVT). 


Notes: Children were not assessed in Year One of the study. To calculate the average the impact of CLI Blueprint in terms of the additional months of growth, RFA used normed scores 
across the age groups of the children in cohort sample. Full statistical models specified as children’s scores nested in centers, with a fixed-effect for year of study and the following 
child-level covariates: age, EL status, and fall PPVT growth scale score. 


Key findings: 


e At the beginning of the year, children in our sample scored similarly in centers receiving CLI 
supports compared to business-as-usual sites (See Table 1, page 6). Spring scores, in contrast, 
were approximately three points higher for children in Blueprint classrooms compared to children 
in statistically comparable business-as-usual classrooms (p<0.05). 


This three-point difference translates to between 2 and 3 months of additional progress in 
vocabulary development in CLI-supported sites compared to children in similar classrooms not 
supported by CLI.” 


Notably, findings were similar across cohorts. 

Figure 2. Receptive vocabulary among children in While many programs see initial dips in outcomes as 
Blueprint and comparison classrooms compared to a result of teachers implementing something new, we 
national average, Spring 2018 and Spring 2019 did not see this post-implementation dip in Blueprint 

classrooms in Year Two. That we did not see additional 
150 ek eps 

gains in Year Three could be because an additional 
145 year of CLI supports does not produce stronger 

outcomes. It could also be because teacher turnover 
140 : : : : 

was quite high, which means that relatively few 
135 teachers received two years of the intervention. This 
430 factor is discussed in more detail in Section 4. 


125 While these results are positive, comparing the 
120 study population scores to national averages is a 
sobering endeavor. For similarly aged children, the 
115 national average PPVT score of 139 is significantly 
higher than the average score of 120 in Blueprint 
110 ‘ : 
centers, reflecting the high poverty rates of our study 
105 sites (discussed in Section 1).*° This means that, 
compared to a national sample of similarly aged 
100 : : ; 
children, the children in our study lag over a year 
= National average (14.8 months) behind their peers in vocabulary 
an ; : growth. As discussed in the introduction, this aligns 
. ae In Blueprint classrooms with the consensus among researchers that, while 
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a Children in comparison classrooms 24 We also tested whether impacts were similar for young children (i.e., those under age 4 
(N=864) by the end of the school year) and for children who were English Learners. We did not find 
Source: PPVT assessments, 2017-18 and 2018-19 evidence of impact differences for younger children or for children whose native language is 
Note: ulndicatce s taretical sicamcance ae pcocslevel torte English compared to their English Learner peers. This may have been because we 
multi-level regression model (see Appendix for more information). pre-screened for English proficiency prior to assessment, thus reducing the sample of EL’s 
Model includes indicators for child age, EL status, and fall PPVT to those with a minimum skill level. Thus, our study does not provide definitive evidence of 
growth scale scores, and assessment cohort. impact for children whose native language is not English. 
25 Ibid. 
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access to high-quality curricula may improve outcomes for children disadvantaged by poverty, to fully 
address the impact of poverty in these communities would require broad-based policy changes to reduce 
or eradicate poverty altogether. 


ll. Teacher perceptions of language and literacy skills and children's 
engagement in classroom activities 


While direct assessments of children’s oral language skills demonstrate positive impact, the benefits of 
CLI’s supports may be more wide-ranging. Though we were unable to assess multiple skills directly and 
in a comparative context, our interviews with Blueprint teachers suggest that there may be additional 
significant positive benefits of Blueprint for children’s outcomes. 


Teachers who used Blueprint materials and approaches to instruction reported that their 
children learned a variety of foundational and higher-level skills for reading and writing 


In Spring 2017 and Spring 2018 interviews with at least one teacher or center director from all 
interviewed centers reported that, through Blueprint, their children were learning foundational and 
writing skills, high-frequency words, how to follow a story, and how to use a physical book. Some 
illustrative examples of what we heard are below. 


e Foundational skills, such as sound awareness, letter recognition, and strong oral language 
skills. “They’re learning... [sound] awareness which ultimately is allowing them to put the sounds 
together to make the word...so it’s really not about the ABC song. They’re really understanding 
what the letters are, what they mean, and how they become a word, and how they become a 
sentence, so I see that intentionality and I see how the children are picking it up.” 


Writing skills and conventions of print, such as punctuation, spaces in between words, the 
direction of print, capital and lowercase letters, and how letters, words, and sentences work 
and work together. “They’re learning how to write a sentence, how a writer writes, how a writer 
brainstorms and comes up with the ideas.” 


New words, both vocabulary words and high-frequency (snap) words. “They’re learning, like, 
cheers they can use, they’re learning snap words and different ways to memorize those words. It’s 
not just, ‘Okay, this is on the word wall’ ‘G-O. Go! What is it? G-O. Go! We’re going to do a cheer to 
go with gol’ So it kind of sticks with them a little bit more.” 


How to follow a story. “They’re learning story progression, they’re learning content, they’re 
learning how to read a book.” 


How to use a physical book. “She sat on the floor, she sat beside me, and she was just talking to 
me, moving her lips. And she was pointing, and she was holding the book the proper way. And she 
was looking at it the proper way and she was turning the pictures and she actually went back and 
looked at different pictures. And she’s only three.” 
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Some Blueprint teachers and center directors also 

attributed the development of higher-level thinking 

skills to Blueprint, including asking questions, talking 

about ideas, comprehending text, making connections, 

predicting, and analyzing themes. One teacher The kids are able to make 

indicated that children who came in with varying connections with thin gS that I 

levels of readiness were exceeding her expectations would never have expected them to 


in terms of applying higher-level thinking: “The kids 
are able to make connections with things that I would learn. Even the ones who Se le 
in the beginning. They get it. 


never have expected them to learn. Even the ones who 
struggle in the beginning. They get it.” - pre-K teacher 


Teachers and directors in Blueprint centers observed children building on this foundation 
by starting to read, write, and sound out words 


In Spring 2017, a few teachers and directors described how Blueprint was helping them instill the 
foundation for children to become successful readers. One director explained, “They’re learning the basic 
foundations of literacy in a whole bunch of areas.” A teacher explained: 


I think they’re learning from Blueprint that vocabulary is very important. Of course, learning 
to read, learning about words, but just knowing that every letter makes a sound, and again, 
letters together make words. I think they’re learning sentence structure, how to read from left 
to right. There’s punctuation marks. Eventually, their reading fluency will be really good. 


A center director explained how the Blueprint advances children’s learning: 


They’re learning awareness which ultimately is allowing them to put the letters—well, the 
sounds together to make the word. They’re really understanding what the letters are, what 
they mean, and how they become a word, and how they become a sentence, so I see that 
intentionality and I see how the children are picking it up. 


Interviewed teachers described how the routines and intentionality of Blueprint supported 
children’s engagement, including the specific elements of Message Time Plus and 
Intentional Read Aloud 


In Spring 2017, teachers and directors from most interviewed centers reported that Blueprint’s routines 
and procedures engaged students. CLI staff and coaches explained that Blueprint was designed to 
promote engagement and exposure to literacy practices, and teachers and administrators reported that 
the structures were effective. Additionally, we heard that Blueprint engages children in intentional and 
focused literacy practices. Teachers and directors noted that children are spending more time reading and 
writing in the classroom and are also more focused, engaged, and answer more questions than they had 
before implementing Blueprint. 
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A teacher explained: 


Every day has an intent, so it’s not like you’re reading the 
book just to be reading it. You have something that you're 


focused on, you’re looking for, youre telling the children, Every day has an intent, so 


“This is what I want you to look for,” or, “I need you to repeat it’s not like you're reading the 
this sentence that’s over and over in the book.” So when it’s : ; : 
book just to be reading it. 


time for you to say the sentence, “Look what they heard him 
say,” they know what they're looking for, and they’re being 
a participant, and also teaching them other literacy skills -pre-K teacher describing 
because the kids literally see it, because you have it written, read-alouds with Blueprint 
“Look what they heard him say.” 


In Spring 2017 and 2018 interviews, teachers, center directors, and coaches described in detail how 
specific elements of Blueprint support children’s engagement: 


Message Time Plus. Teachers especially noted children’s strong engagement with Message Time Plus. 
One center director explained, “Message Time Plus is really beneficial. Just the structure involving the 
students and getting student voices is really beneficial, because it really helps build confidence and helps 
the students to get involved.” Another interviewee described how MTP processes engaged students: 


[It] grabs their attention really quickly, holds them for that time span... It keeps them engaged, 
‘cause they get to come up, and they use the pointer, and they just put the goggles on. It helps 
them to be interactive, which keeps their attention, which is what I think is successful in 
implementing in the classroom. 


Interviewees also reported that children seemed to enjoy working together and helping each other, 
especially during Message Time Plus. One teacher explained, “Sometimes they do like working together. 
They do like helping each other out with identifying letters. If a child gets it wrong, I’ll ask them to call on 
a friend to help them.” 


Intentional Read Aloud. Some coaches and teachers said that, since classrooms adopted Blueprint, 
children are more engaged in stories. One coach explained, “They just seem so excited about the books, 
because we read [the same] books sometimes two or three times in a week.” One teacher reported that 
children in her classroom act as the teacher during center time and reenact read-alouds. The teacher said, 
“I’m just listening and she’s like, ‘Well, this is the front cover of the book, and this is the back cover, and 
this is the title page. Don’t you like the title page? Look how colorful it is. ... I’d say they’re more engaged 
with the reading.” 


Teachers and directors reported that Blueprint’s positive engagement resulted in an 
increased love of reading and writing, self-esteem, and parental engagement in literacy 
practices 


In Spring 2017, we heard that children were learning the joys of reading and writing through Blueprint. 

A few teachers and directors said Blueprint supported an increased interest in reading and writing 
(including reading more often) and a greater engagement in the read-alouds since implementing Blueprint 
in their classrooms. Additionally, two directors reported that this exposure and engagement in intentional 
literacy activities led to signs of children’s heightened self-esteem. These directors reported that children 
were excited to engage and share their new knowledge as their comfort with literacy practices increased. 
One center director explained that, “The fact that they can actually read a word and then a sentence at this 
level—they’re really excited.” Another director spoke of children’s excitement about their growth, saying 
“They are learning to expand their sentences to be able to look at words and actually read it, want to do it, 
want to be engaged. They want to express what they learn, which is a great thing. I think their self-esteem 
is building because they are learning so much, and they’re excited about it.” 
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B. Impact on Teacher Practice and Classroom Environments 


CLI expects their supports will help teachers become more intentional and knowledgeable about early 
literacy content and pedagogy and develop enhanced skills in the management of classrooms and the 
creation of a literacy rich environment. Below, we assess the impact of CLI’s pre-K language and literacy 
supports on teachers’ perceived knowledge and ability to implement effective practices for early literacy 
instruction. We also examined differences in Blueprint and comparison centers in the self-reported 
frequency of specific instructional practices for teaching reading, oral language, and writing skills. 


Key Findings: Teacher Knowledge and Instructional Practice 


e Relative to teachers in comparison centers, more surveyed Blueprint teachers reported 
considerable growth in ability to implement effective practices for early literacy instruction 
from fall 2018 to spring 2019. Smaller differences in teacher knowledge were evident but 
not statistically significant. 


Most surveyed teachers in spring 2019 reported that Blueprint is very useful for supporting 
instructional practices, both specific to language and literacy and more generally for 
creating a positive environment. 


In surveys, more Blueprint teachers reported frequently implemented some, but not all, 
best practices for instructing beginning reading and oral language skills, compared to 
teachers in business-as-usual classrooms. 


|. Teacher Perceptions of Knowledge and Ability to Implement Best Practices in 
Language and Literacy Instruction 


At the end of Year Three of our study, we surveyed lead teachers in Blueprint and comparison classrooms 
about their perceived knowledge of and ability to implement effective instructional practices for early 
literacy. We then asked teachers to reflect on growth in their knowledge and ability to implement over the 
course of the school year and, for Blueprint teachers, if they felt CLI supports contributed to their growth. 
Because these data are self-reported perceptions, we interpret differences between Blueprint teachers 
and their peers as suggestive. Further research, for example drawing on direct observations of instruction 
and/or assessments of specific knowledge domains, would strengthen these findings. 
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Blueprint teachers reported higher levels of knowledge and stronger ability to implement 
effective instructional practices relative to teachers in comparison centers 


Figure 3 shows the percent of lead teachers in Blueprint and comparison centers reporting strong 
knowledge and ability to implement effective instructional practices for teaching language and literacy in 


Spring 2019. 


Figure 3. Percent of lead teachers in Blueprint and comparison centers reporting strong knowledge and ability to 
implement effective instructional practices for teaching language and literacy in Spring 2019 


% of Lead Teachers 


Teacher feels "very" Teacher feels "very" 
or "extremely" or "extremely" 
knowledgable about able to implement 
effective practices effective practices 


= Blueprint lead teachers = Comparison lead teachers 
(N=27) (N=28) 


Source: RFA teacher survey, 2018-19 


Note: * indicates statistical significance at p<0.05 level from a two-sample test of proportions. Lead teachers are defined as those CLI targeted for 
coaching in Blueprint centers. In comparison centers, lead teachers are those with responsibility for leading instruction. 


Findings: 


e CLI’s supports for pre-K may improve teacher ability to implement effective instructional practices 
for teaching pre-K children language and literacy. Nearly all (96%) of Blueprint teachers reported 
feeling able to implement effective instructional practices, compared to 61% of teachers in 
comparison centers, a statistically significant difference (p<0.001). 


Similarly, CLI may improve teacher knowledge of effective practices. Nearly all (89%) of Blueprint 
lead teachers reported high levels of knowledge about effective practices, compared to 68% of 
lead teachers in comparison centers (p<0.101). Though not statistically significant, this difference 
is substantively meaningful. 
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Twice as many Blueprint teachers reported considerable growth in ability to implement 
effective instructional practices from fall to spring 
We asked teachers to reflect on their growth in knowledge and ability to implement effective instructional 


practices throughout the 2018-19 school year. We also asked Blueprint lead teachers to report if they felt 
their improvements could be attributed to the CLI professional development services they received. 


Figure 4. Percent of lead teachers in Blueprint and comparison centers reporting considerable growth in 
knowledge and ability to implement effective practices for early language and literacy instruction from Fall 2018 


to Spring 2019 

100% 
90% 
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40% 


% of Lead Teachers 


30% 
20% 
10% 


0% 
Teacher knowledge of CLI contributed to Teacher ability to CLI contributed to 
effective practices has growth in knowledge implement growth in ability to 
improved "quite a bit" "quite a bit" effective practices has implement "quite a bit" 
or "a great deal" or "a great deal" improved "quite a bit" or "a great deal" 
or"a great deal" 


= Blueprint lead teachers = Comparison lead teachers 
(N=27) (N=28) 
Source: RFA teacher survey, 2018-19 


Note: * indicates statistical significance at p<0.05 level from a two-sample test of proportions. Lead teachers are defined as those CLI targeted for 
coaching in Blueprint centers. In comparison centers, lead teachers are those with resposibility for leading instruction. 


Specifically: 


e More Blueprint teachers reported considerable improvements in knowledge and ability to 
implement effective instructional practices. Two-thirds (67%) of Blueprint lead teachers reported 
considerable growth in knowledge of best instructional practices from fall to spring, compared 
to half (50%) of lead teachers in comparison centers. Though this difference was not statistically 
significant (p<0.227), the size of the difference between groups is substantively meaningful. 


More than twice as many (81%) Blueprint lead teachers as comparison lead teachers (39%) 
reported considerable growth in their ability to implement effective practices from fall to spring, a 
statistically significant difference (p<0.001). 


Nearly all Blueprint teachers (85%) who reported growth in their knowledge and skill reported 
that CLI contributed either “quite a bit” or “a great deal” to their growth. 
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ll. Frequency of Implementing Best Practices for Instructing Early Language and 
Literacy 


We surveyed Blueprint teachers to determine the extent to which teachers frequently implemented 
effective language and literacy instructional practices.”° Teachers were asked how many times in the 
past month their class engaged in specific language and literacy activities. We compared the percent 

of teachers that reported frequently implementing each activity, defined by engaging their class in the 
activity at least 3-4 times per week in the last month. We examined reading, oral language, and writing 
activities. Results are presented for each below (Tables 3-5). We ordered activities by the percent of 
comparison teachers that reported frequently engaging in each activity, which gives a sense from left to 
right of business as usual. 


More Blueprint teachers frequently implemented some best practices for instructing 
beginning reading and oral language skills, but not writing skills 


In Figure 5, we present teacher-reported frequency of reading activities, such as listening to the teacher 
read a book or working on phonics. 


Figure 5. Percent of lead teachers in Blueprint and business-as-usual centers reporting frequently implementing 
various instructional activities for early reading, Spring 2019 


Children in your class... nati 


% of teachers implementing activity names of letters 


3-4 times per week or more . 


Learn about 
conventions of 
print 


Read or 
pretend to read 
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book phonics 


mi % of teachers implementing business-as-usual (N=28) Source: RFA teacher survey, 2018-19 
Note: Lead teachers are defined as those CLI 


ie % of teachers implementing Blueprint (N=27) targeted for coaching in Blueprint centers. In 
comparison centers, lead teachers are those 
More Blueprint teachers implementing frequently than with responsibility for leading instruction. 
business-as-usual 
* indicates statistical significance at p<0.05 
More Blueprint teachers implementing frequently than level from a two-sample test of proportions. 
business-as-usual 


26 This list was generated in partnership with CLI and drew on a national survey developed by NCES (Early Childhood Longitudinal Study-Birth Cohort). 
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Notable findings: 


e While there are some similarities in Blueprint and comparison classrooms (e.g., nearly all 
teachers, regardless of group, frequently have their class listen to them read a book), there are 
also important differences that suggest that more teachers in Blueprint classrooms frequently 
implement some best practices in reading instruction. 


Notably, for reading activities, more Blueprint teachers frequently implemented practices that, 
in business-as-usual condition, are not consistently implemented with high frequency: teaching 
about conventions of print and working on phonics. Compared to less than 60% of comparison 
teachers, 85% of Blueprint teachers reported frequently teaching their class about conventions 
of print (p<0.05), and compared to only 35% of comparison teachers, two-thirds of Blueprint 
teachers reported frequently having their class work on phonics (p<0.05). 


We also asked teachers to reflect on the frequency of oral language activities, such as singing songs and 
discussing new words. Results are presented in Figure 6. 


Figure 6. Percent of teachers reporting frequently implementing various instructional activities for oral language, 
Spring 2019 
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Source: RFA teacher survey, 2018-19 
a % of teachers implementing Blueprint (N=27) Note: Lead teachers are defined as those CLI 
targeted for coaching in Blueprint centers. In 
More Blueprint teachers implementing frequently than comparison centers, lead teachers are those 
business-as-usual with responsibility for leading instruction. 


© Blueprint and business-as-usual teachers are not 
substantively different from each other 
Notable findings: 


e There were no statistically significant differences between comparison and Blueprint teachers. 
Yet across all indicators, the pattern of responses suggests that a larger percent of teachers in 
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Blueprint classrooms were frequently implementing oral language activities. 


More Blueprint teachers reported frequently discussing new words than did comparison 
teachers (74% vs 50%), though this difference is not statistically significant. Similarly, though 
not statistically significant, more Blueprint teachers reported frequently working on phonemics 
(41%) and word families (31%) than did comparison teachers (29% and 18%, respectively). 


We asked teachers to report how frequently they use a variety of instructional activities to support early 
writing. Results are summarized in Figure 7. 


Figure 7. Percent of lead teachers in Blueprint and business-as-usual centers reporting frequently implementing 
various instructional activities for early writing, Spring 2019 


Children in your class... 


% of teachers implementing activity 
3-4 times per week or more 


Share and 
discuss their 
own writing 


Generate ideas 
for writing 


Practice Practice 
writing writing 
their name the letters of the 
alphabet 


Source: RFA teacher survey, 2018-19 
Note: Lead teachers are defined as those CLI 


a % of teachers implementing Blueprint (N=27) targeted for coaching in Blueprint centers. In 
comparison centers, lead teachers are those 
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@ Blueprint and business-as-usual teachers are not 
substantively different from each other 


Notable findings: 


Compared to other language and literacy activities, fewer teachers in both Blueprint and 
comparison classrooms frequently implemented best practices for instructing beginning writing 
skills. 


Unlike in reading and oral language activities, we see evidence for writing activities that more 
teachers in comparison classrooms are implementing some best practices for writing. More 
teachers in comparison classrooms reported that children in their classrooms practiced writing 
their own name (64%) and sharing their ideas in their own writing (54%), compared to Blueprint 
teachers (56% and 41%, respectively). These differences are substantively meaningful, but not 
statistically significant. 
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Ill. Utility of Blueprint for Supporting Teacher Instructional Practice 


Most Blueprint teachers reported that Blueprint is very useful for supporting instructional 
practices 


In 2019, we surveyed teachers in CLI-supported classrooms about their perceptions of the utility of the 
Blueprint for Early Literacy curriculum for supporting teacher instructional practices. 


Figure 8. Percent of lead teachers who report that Blueprint is “very” or “extremely” useful, Spring 2019, N=27 
teachers 
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Source: RFA Teacher Survey, 2018-19 


Note: RFA defines “lead” teacher as the CLI teacher targeted to receive coaching 
Findings: 


Most surveyed teachers reported that Blueprint is “very” or “extremely” useful for supporting a 
variety of instructional practices, including planning a variety of engaging language and literacy 
activities (93%), identifying when children are struggling with their language and literacy 
development (85%), creating a positive learning environment (81%), and managing classroom 
behavior (70%). 


In Spring 2018 interviews, we asked the eight interviewed Blueprint teachers to reflect on what 
specifically they were learning from CLI supports and the Blueprint curriculum. Teachers provided 
examples of what they learned: 


e Information about the foundations of literacy. One teacher reported learning about the role 
that phonemic awareness plays in early literacy development. 


To set high expectations for students. One teacher reported learning that setting high 
expectations contributes to strengthening children’s confidence. Another teacher reported that, 
before using Blueprint, instruction was mostly focused on foundational literacy skills. Since 
introducing Message Time Plus, this teacher realized that students were ready to dive deeper into 
literacy and meaning-making. The teacher explained, “Before we started taking the CLI training, 
it was more like, you know, phonics, phonemic awareness, sounds, and rhyming words and things 
like that were more important than sentences, how to build a sentence, sentence structure. You 
would think that'll come more in the [later] grades. But seeing them utilize it [during Message 
Time Plus]—and the confidence it gives them when they do learn that—that’s really important as 
well.” 
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To make instruction more intentional and adapt it, when necessary. A teacher said, 
“[Blueprint] really helped me dive into my goals and objectives...If you prepare before the kids 
get there, then you'll have a more rich experience.” Another teacher described learning how to be 
a more intentional teacher through CLI training and coaching: “I know what I’m teaching. I know 
what I need to work on. I know my goals. I know my outcomes.” This teacher explained that she 
learned to adapt her lessons extemporaneously because she became so aware of her specific, 
intentional instructional goals. 


To make instruction engaging. Blueprint taught a teacher that “if you’re not excited, [your 
students] won't be excited about something.” This teacher credited CLI coaching with adding tools 
like using movement to teach “snap” (high-frequency) words to make instruction more engaging. 
Another teacher reported learning “how to get the kids really interested in the books and stuff 
using my personality while reading the story.” 


Underlying these improvements may be an increase in teacher confidence. Half of the teachers 
interviewed in Spring 2018 reported that Blueprint increased their confidence and ability to be effective 
educators. One CLI staff person and one center director agreed that they saw teacher confidence 
improving throughout the intervention. A teacher confirmed this staff member’s theory, explaining, “It 
just builds your confidence to come in. After you finish implementing [Blueprint] and you see the results, 
it’s like, ‘I have more confidence in doing this. I want to keep doing this.’ I feel like that’s what’s really good 


about it.” 
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CLI also supports teachers with specific strategies to make their classrooms more literacy-rich, positing 
that a strong language and literacy environment will promote children’s learning and engagement. To 
understand CLI’s impact on classroom language and literacy environment, RFA observed Blueprint 

and comparison classrooms using the Early Language and Literacy Classroom Observation, Pre-K tool 
(ELLCO) in the spring of 2018 and 2019. In this section, we present results from data pooled across study 
Years Two and Three. More about the observation process and inter-rater reliability of RFA’s observations 
can be found in the Appendix. 


The ELLCO provides data on two aspects of the classroom environment: 


The general classroom environment. This ELLCO subscale consists of two 
components: classroom structure and curriculum. Classroom structure addresses 
classroom organization and contents, children’s access to and use of materials, 
management practices, and adult roles and professional focus. Curriculum consists 
of the curriculum environment, instructional strategies, opportunities for child 
choice and initiative, and recognition of diversity in the classroom. 


The language and literacy environment. This subscale consists of three 
components: language environment, books and book reading, and print and early 
writing. Language environment addresses discourse climate in the classroom, 
opportunities for extended conversations, vocabulary development, and efforts 

to develop phonological awareness. Books and book reading addresses the 
organization and use of the book area, the characteristics of books available, 

the presence and use of books across content areas of the curriculum, and the 
quality and frequency of book reading. Finally, print and early writing includes the 
availability of writing materials, opportunities that build awareness of print and 
varied purposes of writing, instructional strategies, and use of environmental print. 
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Scores for each scale range from 1 to 5. To analyze the impact of Blueprint on the classroom environment, 
we compared average scores in Blueprint and comparison classrooms. 


A note about interpreting the size of effects. To examine the magnitude of impact, we also discuss the 
average differences between Blueprint and comparison classrooms in terms of effect sizes (we also use 
this nomenclature in the next section on Blueprint impacts for children’s vocabulary). An effect size takes 
the difference in averages across groups relative to the known population variation in the scale (i.e., the 
population standard deviation). For classroom-level outcomes, an effect size of .8 is generally considered 
large, .5 is medium, and .2 is small. 


Blueprint classrooms scored moderately higher on measures of classroom literacy 
environments than did comparison classrooms 


Below, in Figure 9, we present the average scores on the five-point ELLCO scale for Blueprint and 
comparison classrooms and describe differences in terms of effect sizes. 


Figure 9. Classroom environments of Blueprint and comparison classrooms, Spring 2018 and Spring 2019 
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Source: RFA observations of classrooms using ELLCO Pre-K tool; 2017-18 and 2018-19 


Note: * indicates statistical significance at p<0.05 level from a t-test of mean differences. 


Findings: 


e On all measures of classroom language and literacy environments, CLI-supported classrooms 
scored higher than comparison classrooms. Though all differences represent small-to-medium 
effect sizes, differences are statistically significant for literacy subscales only: Books and book 
reading and Print and early writing. The effect sizes books and book reading (50%) and print and 
early writing (43%) are moderately-sized, statistically significant effects (at the p<0.05 level.) 


Differences in measures of classroom structure, curriculum, and language environments in CLI 
supported classrooms and comparison classrooms translate to small effect sizes: A 0.10-scale- 
point difference between average scores for CLI- and non-CLI supported classrooms translates to 
20%, 14%, and 25% of a standard deviation for classroom structure, curriculum, and language 
environment, respectively. These differences are substatively meaningful, but not statistically 
significant. 
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Section 4: Implementation of CLI’s Supports for pre-K 
Language and Literacy in Philadelphia 


CLI’s range of positive impacts for children, teachers, and classrooms is particularly notable because, as 
our implementation research found, there are many challenges to providing multiple years of professional 
development and implementing a new curriculum supplement in early childhood educational 
environments. In this section, we explore the extent to which teachers received the intended amount of 
support from CLI and discuss both challenges and successes of implementation. 


This report section is organized into three sub-sections: 


A. Teacher turnover. One of the biggest challenges of providing professional development 
support throughout the study period was the high level of teacher turnover. In this section, we 
describe the teacher staffing trends in Blueprint classrooms and review the implications for 
implementation of CLI’s Blueprint. We also present teacher perspectives on what it takes to 
retain pre-K teachers, in a “spotlight on teacher turnover.” 


. Fidelity of CLI’s Professional Development. This section examines the extent and quality of 
CLI’s training and coaching, including an in-depth description of coaching goals, key elements, 
focal areas for support; approaches to integrating assistant teachers in coaching sessions; and 
teacher-reported benefits of and satisfaction with CLI coaching. 


Quality of Blueprint for Early Literacy implementation in the classroom. This section 
examines Blueprint implementation in the classroom, focusing on teacher fidelity of 
implementation of key Blueprint elements: Message Time Plus, Intentional Read Aloud, and 
Power of Three. We also provide descriptions of the instructional roles of lead and assistant 
teachers in their pre-K classrooms and discuss implications for supporting classroom 
instructional environments. 


A. Lead Teacher Turnover in Blueprint Classrooms 


RFA tracked lead teacher staffing in CLI-supported classrooms during the spring of each study year. As 
is typical of pre-K environments more generally, there were substantial over-time shifts in lead teacher 
staffing during this 3-year study period, and teacher attrition in study classrooms was substantial. 


Only 40% of lead teachers from Year One of the study remained in their positions by the 
end of Year Three 


As Figure 8 below shows, by the end of Year One in Spring 2017, there were 33 classrooms receiving 
supports from CLI, each with at least one teacher identified by CLI to receive Blueprint coaching and 
training. By the end of Year Two in Spring 2018, only 19 (or 58%) of these lead teachers remained in their 
positions. By the end of Year Three in Spring 2019, only 14 teachers (or 42%) who were enrolled in the 
study in Year One remained in their positions. 
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Figure 10. Number of lead teachers remaining in their roles from Spring 2017 to Spring 2019 
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Source: CLI administrative data, 2016-17, 2017-18, and 2018-19 


Note: RFA defines “lead” teacher as the teacher CLI targeted to receive coaching. 


Centers opened and closed classrooms throughout the three-year study period as well. By Year 
Three of the study, there were 36 classrooms operating in centers receiving CLI supports. Given teacher 
turnover rates, this meant that only 14 of the 36 classrooms had teachers leading instruction that had 
been involved in the project from the start (just about 40%). 


The substantial staffing shifts over time resulted in an evolving target for CLI during their implementation 
of their supports for pre-K language and literacy. We discuss the implications of the significant staffing 
challenges throughout this section. 
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Spotlight on 
Teacher Turnover 


Annual pre-K teacher turnover is estimated at 

30% nationally. The pre-K centers in this study 
experienced teacher turnover at rates higher than 
national estimates--an average of nearly 50% per 
year. Here we present teacher perceptions from our 
interviews with teachers in spring 2018 of what is 
needed to support pre-K teacher retention. 


Teacher feedback regarding retention: 


e Compensation—in wages and benefits—should support the cost of living. Half of 
interviewed teachers mentioned the importance of being adequately compensated for their 
work. One teacher from a site with low turnover explained that a key reason that teachers 
continue to work in or leave pre-K sites is how much they are paid: “No matter how much 
you like your job, you still have to live.” Another teacher, also from a site with low turnover, 
added that benefits are key factors in determining whether to stay at a site or leave: “I came 
from [another] preschool, and I couldn't stay... I didn’t have good benefits, I was paying a lot 
of money for my benefits.” 


Professional growth should be supported. Multiple teachers who worked in centers 
where they received support for their professional growth named this support as key 
to retention. One teacher reported that teachers want to continue working at sites that 
are “challenging you to grow and providing you with the tools.” Another teacher agreed, 
explaining that their center director encouraged them to get a bachelor’s degree, which 
enabled them to become a lead teacher. 


Teachers’ work in the classroom should be supported. Most interviewed teachers 
mentioned the need to feel support from center management. A teacher from a site with 
high turnover explained, “I think a lot of teachers feel like they don’t have the support from 
management that they need.” Other teachers from sites with lower turnover rates pointed 
to support from the center administration or their colleagues as essential. One of these 
teachers highlighted the administration’s support “communicating with parents [and] 
helping with the material that’s needed in the classroom” as important, while another 
teacher identified a “wonderful” teaching partner as a key reason for continuing to work at 
her site for many years. 


Teachers should feel valued. One school district teacher described how critical it is to be 
valued for their work: “You know [as pre-K teachers,] we're still doing as much work as the 
grade teachers, if not more, because the students are very needy, and they’re just learning 
these things for the first time. So, it’s really important that you’re valued on the same level 
as the grade teachers.” 
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CLI provides lead pre-K teachers with training workshops on evidence-based practices for early literacy 
instruction and in-person, content-focused coaching to support their implementation of Blueprint for 
Early Literacy. In this section, we report training attendance and coaching hours received during the 
three-year study period. 


Key Findings: Blueprint Training and Coaching and Dosage 


Fewer than one in five lead teachers received the full amount of intended training and 
coaching supports by the end of Year Three. However, almost three in four received at least 
a basic level of support, defined as participating in an introductory training and receiving 
20 hours of coaching. 


Although high turnover and number of allotted coaching hours posed challenges for ideal 
CLI coaching, coaches implemented various strategies to maximize their support, like inte- 
grating assistant teachers into coaching when possible. 


Three in four lead teachers surveyed in 2019 reported that coaching sessions included 
pre- and post-conferences at least most of the time, though coaches reported that 
conferences were often shorter than ideal, and, because they occurred in the 
classroom, teachers’ ability to focus was often inhibited. 


Despite challenges, teachers reported that they were very satisfied with the training and 
coaching they received, felt coaching prepared them to implement the Blueprint curriculum, 
and viewed the CLI’s professional development as a substantial benefit. 
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|. Training Attendance and Coaching Hours Received 


In the following sections, we describe the extent of CLI training and coaching received among lead 
teachers who were working in classrooms at the end of the study, in Spring 2019. 


By the end of the study, just two in five lead teachers had attended all three core CLI 
trainings 


We used data from CLI’s administrative records to assess how many lead teachers attended the three core 
CLI trainings by the end of Year Three (Figure 11). 


Figure 11. Percent of lead teachers who attended core CLI trainings from 2017-2019 
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Source: CLI administrative data, 2016-17, 2017-18, and 2018-19 


Note: Retained lead teachers are those that have been working as leads in centers since Spring 2017. RFA defines “lead” teacher as the teacher 
CLI targeted to receive coaching. 


Findings: 


e Ofthe 36 lead teachers working in Blueprint centers in Spring 2019, only about 40% had attended 
all three core trainings. A majority (78%) attended an introductory training, which CLI offered 
at least once each year. Fewer lead teachers attended IRA and MTP trainings—61% and 47% 
respectively. 


Training attendance was higher for teachers that were in their positions at the start of the study in 
Spring 2017: about two-thirds (64%) of retained teachers attended all three trainings. 


We investigated why training attendance, particularly for MTP and IRA training was relatively low, even 
for retained teachers. Several teachers and directors reported time conflicts and the need to use the 
weekend to recuperate after a long week kept them from attending weekend trainings and meetings. IRA 
and MTP trainings were offered less frequently than the introductory training, so newer teachers and 
teachers who had scheduling conflicts had fewer opportunities to make up those trainings. 
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By the end of the study, less than 30% of lead teachers had received the full amount of CLI 
coaching 


As with the low training rates discussed above, the high rate of teacher turnover limited the extent 
to which teachers working in centers in spring 2019 had received the intended 60 hours of coaching 
cumulated over the three years of the study (Figure 12). 


Figure 12. Percent of lead Blueprint teachers by hours of coaching received from 2017-2019 
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Note: Retained lead teachers are those that have been working as leads in centers since Spring 2017. RFA defines “lead” teacher as the teacher 
CLI targeted for coaching. 


Findings: 


e Atthe end of Year Three, only 28% of lead teachers had received at least 60 hours of coaching, or 
at least 20 hours annually for three years. 


As with training attendance, coaching received was higher among retained lead teachers: All 
retained leads received at least 40 hours of coaching, and 71% of retained lead teachers received 
at least the full 60 hours of CLI coaching. 


Boosting Children’s Language and Literacy 
RESEARCH for ACTION Skills through Blueprint 


Because of high turnover, less than 20% of lead teachers received the full dosage of CLI 
training and coaching by Year Three 

Overall, CLI aimed to provide the full suite of supports for lead teachers in 11 Philadelphia pre-K centers 
by the end of Year Three, including three core trainings and 60 hours of in-person coaching. In Figure 
11, we show the extent to which, by the end of the study, teachers working in CLI-supported centers had 


received a basic level of support (20 hours of coaching and an introductory training) and the full suite of 
CLI supports. 


Figure 13. Percent of lead Blueprint teachers by level of CLI supports received from 2017-2019 
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Note: Retained lead teachers are those that have been working as leads in centers since Spring 2017. RFA defines “lead” teacher as the CLI 
teacher targeted to receive coaching. 


Findings: 


e Only 17% of lead teachers in Blueprint classrooms in Spring 2019 received the full intended 
amount of training and coaching. 


However, a majority—72%—had received a basic level of support, at least attending the 
introductory training and receiving at least one full year’s worth of coaching (20 or more hours). 


It was difficult for CLI to deliver the full suite of services even for teachers that remained in 
their roles over time, in part because of teacher schedules and availability of makeup trainings 
discussed above. Though most (86%) received at least a basic level of support, less than half of 
retained teachers (43%) had received the full suite of CLI supports by the end of Year Three. 
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Il. Successes of CLI Training and Coaching 


Over 80% of surveyed teachers reported that CLI professional development helped them 
feel prepared to implement Blueprint for Early Literacy 


In spring 2019, we surveyed teachers about the extent to which the CLI professional development 


components prepared them to implement the Blueprint for Early Literacy. Results are presented in Table 
4, below. 


Table 4. Surveyed teacher perceptions of the extent to which CLI trainings and coaching prepared them to 
implement Blueprint for Early Literacy, Spring 2019 


| | Of the 36 lead teachers in CLI-supported classrooms in spring 2019.... 


# of Teachers # of Teachers that % of surveyed teachers 
Intervention Participated in Responded to Spring 2019 reporting component prepared 
Component Intervention Component Teacher Survey them to implement Blueprint 
Introduction to 0 
BP training 28 86% 


IRA training? 17 100% 
83% 
87% 


MIP training 22 
Coaching 26 


Source: RFA Teacher Survey, 2018-19 


*Data for IRA training come from Spring 2018 survey because CLI did not offer an IRA training in Year Three. 
Findings: 


Over 80% of surveyed teachers felt the trainings and coaching better prepared them to implement 
Blueprint for Early Literacy. 


In spring 2017 interviews, a few teachers described how Blueprint trainings helped familiarize them with 
materials and approaches to using Blueprint in the classroom. One teacher explained how the training 
allowed her time and space to become comfortable with Blueprint: 


I had looked in the book, but I didn’t really have a firm foundation of what everything was... 
what was the purpose of it. And she [the trainer] really went through that book tooth and nail. 
We had a treasure hunt search with the book, so it allowed me time to look at it, not being at 
work, and I could really focus on [it]. 


According to a few teachers, networking and collaborating at trainings also provided opportunities to 
problem-solve and learn by listening to other teachers’ experiences. One teacher said, “It [training] helped 
me out so much because | heard from other teachers how it works, when they do it, what they do.” 


Teachers reported benefitting from different elements of coaching support 


In Spring 2017 interviews, teachers offered examples of how different elements of coaching were 
beneficial, including: 


e Modeling. Teachers described coach modeling as helpful in seeing how Blueprint should look 
in action. One teacher described how a coach used a gradual release model to support her in 
adopting new practices: “The first time she came, I was explaining to her my difficulties; some of 
these kids have never sat in a circle, and some of them are advanced, and she did the lesson for 
me. Then the next time—I did one part and she did the other part. Then the last time she came, | 
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did both parts.” Another teacher explained that modeling supported her learning style. She said, 
“It helped me have a better insight on how it looks. I could do it, but if | saw her do it then it made 
sense. I’m a visual learner.” 


Feedback. Teachers said that the feedback coaches provided after observing their lessons helped 
them strengthen Blueprint implementation. One teacher described how a coach helped her to 
implement MTP using a range of strategies, including supplying materials, discussion, modeling 
it and recording a lesson for coach-teacher discussion. She noted, “Accepting her criticism and 
feedback is also good.” 


Guidance with classroom environment and behavior management. One teacher described 
how her coach helped her with multiple practical tasks, including putting up posters about the 
Power of Three and setting up the library. The coach also “did a lot of observations because I had 
a lot of behavior issues. She helped me. Everybody had to go to the bathroom, things like that, 

she helped me manage that kind of stuff in order to actually make my message time longer.” This 
teacher saw direct links between coach support with classroom management and enhanced focus 
on instruction. 


Troubleshooting specific challenges. Coaches also helped teachers address specific challenges 
they encountered as they implemented Blueprint, such as differentiating lessons for a range of 
students. One teacher said, “I have two kids reading already. She helped me to build on what they 
already know, so they don’t get bored. She provides activities for them. As well, some children, 
they’re struggling a little bit...they still need certain skills. She helps to give us activities or 
strategies to help them.” 


Encouragement. Adopting new classroom approaches can be daunting. One teacher described 
her coach telling her, “You’re doing a good job. Don’t be nervous.” 


lll. Challenges for CLI Training and Coaching 


Coaches and CLI staff reported that training and coaching in the context of high teacher 
turnover was a significant challenge 


Training and coaching within the context of high teacher turnover posed major challenges for CLI. First, 
turnover required coaches to essentially ‘start over’ with new teachers, trying to bring them up-to-speed 
even though they missed the trainings. When this happened, coaches also needed to begin again with 
building the relationships which serve as the foundation for coaching. 


As one coach said: 


The thing that is worrisome to me is there’s 
so much turnover. It seems like you just get a 
The thing that is worrisome to me teacher who’s rolling, who’s gelling, and then... 
is there’s so much turnover. Yesterday, a teacher said to me, “Oh, I just got 
a new job.” I said, “Are you kidding me? I just 
started with you.” I had maybe eight hours in 
-CLI coach and we were hitting the ground running. 
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In interviews with CLI staff and coaches, we probed deeper into this challenge. We found that it is difficult 
to productively coach new teachers before they receive training. CLI reported that they typically consider 
the number of new teachers that need training and the time of year that teachers start when deciding to 
have a make-up training for new teachers. During our study, not all new teachers were able to attend a 
training before they receive coaching. All four coaches reported that coaching new teachers before they 
have attended trainings is a challenge. One coach described new teachers’ understanding of Blueprint 

as “kind of foggy until they attended the training.” This coach continued to explain the importance of 
attending trainings to fully benefit from coaching: 


When the teachers go for training, it’s usually a five-hour training. When the coaches come 
into the classroom, we can stay no longer than between an hour to three hours, so we can’t do 
everything in an hour, so it’s really imperative that they attend the five-hour training. If they 
actively attend the five-hour training, a light bulb goes on to say, “Oh, this is what [my coach] 
meant.” 


Finding adequate time for coach/teacher pre- and post-conferences was a common 
challenge across centers 


One of the primary challenges to productive coaching was finding time for coach-teacher conferences 
during coaching sessions. Generally, pre-K teachers have no dedicated planning time or free periods. 
Moreover, sometimes teachers do not have an assistant in the room with them when the coach visits. In 
Year Two of the study, directors made more of an effort to ensure that teachers had time for coaching 
conferences, according to some coaches. However, overall, coaches and teachers continued to find it 
logistically challenging to find the time to conference during their coaching sessions. In Spring 2017 
interviews, one teacher described the difficulty of having focused one-on-one meetings with the coach 
under such circumstances: 


I just think that it should be sometimes away from the children. Sometimes, when [the coach] 
comes, there’s nobody to cover the space so that we can really talk about it and analyze it. It 
doesn’t take long sometimes, but I know one time, we had to do it inside of the classroom. I 
heard what she was saying, but you're making sure Johnny over here is doing the right thing in 
this center. 


Nonetheless coaches, teachers, and directors made it work. When surveyed in spring 2019, most 
teachers reported that they had pre- and post-conferences either “most of the time” or “always” (Figure 
14), though about one in five teachers reported that they only were able to find time for pre- and post- 
conference “sometimes.” 
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Figure 14. Percent of lead teachers reporting the frequency of coaching pre- and post-conferences during 
coaching sessions, Spring 2019, N=26 
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Source: RFA Teacher Survey, 2018-19 


Note: RFA defines “lead” teacher as the teacher CLI targeted to receive coaching. 


Challenges related to the pre-K context can reduce the quality of coach-teacher 
conferences 


Though strategies were used effectively to make coaching conferences happen for the most part, we 
observed nuanced challenges to the quality of coach-teacher conferences related to time constraints and a 
lack of ability to maintain focus when conferences occur in the classroom in our spring 2019 interviews. 


Often, time constraints made pre- and post-conferences shorter than ideal. Two coaches reported that 
pre- and post-conferences were often shorter than ideal. “The [pre-conference] is usually shorter than 

I would like it to be. It’s hard to pull these teachers, especially early in the morning. They’ve got parents 
and breakfast and a lot going on. It could be as little as ten minutes,” one coach explained. She added, “We 
don’t always get to the post-conference, that’s for sure.” Another coach said that, although sometimes 

pre- and post- conferences lasted the ideal 15 to 20 minutes, there were times when they were both five 
minutes long. 


In-class conferences inhibited teachers’ ability to focus on coaching content. Unlike in K-3 settings, 
pre-K teachers rarely have time each week outside of the classroom to develop their instructional 
practice. As a result of this limitation, CLI coaches often must conduct coach conferences in the classroom, 
while teachers are still supervising children. In interviews, coaches and teachers reported in-classroom 
conferences can inhibit teachers’ ability to completely focus on coaching content. One lead teacher 
explained that it was difficult to focus during these coaching conferences: 


I’m taking my eyes off the children. So, that could be a little difficult. Trying to talk to [my 
coach] and talk to [the children]. Because sometimes when I’m talking to [my coach], [the 
children will] call me and then I have to redirect myself in all these different areas. So, that part 
is a little tricky. 
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Another lead teacher explained: 


I wish that we had more one-on-one time...the only time I can do it is lunch break and there’s 
two teachers so you can only get a half hour. I wish that we just had more time to reflect 
without being in the classroom when the kids are there. 


All four CLI coaches also reported that in-the-classroom conferences posed a challenge to successful 
coaching. “I think we have their full attention outside of the classroom,” one coach explained. “When 
they are in the classroom, they may miss something here or there, or misinterpret it, just because of the 
distractions, and may not always tell us until we observe something during the next visit.” Another coach 
explained her perspective regarding the way it impacted the quality of coaching, noting that the lack of 
focus during in-class conferences can limit the depth of the coaching she is able to provide: 


It definitely affects [the quality of the coaching], depending on the teacher and how well they’re 
able to focus on the conversation and not be pulled by a behavior over here, or the classroom is 
too loud, or kids are coming over and interrupting. Some teachers are completely able to talk 

with me, but it’s really hard. I would say the majority struggle with that. They can’t quite focus. 


This coach explained that she tried to mitigate the challenge by talking fast to get as much information in 
as she could, prioritizing and making sure she said the things she needs to say, and occasionally meeting 
with teachers during their lunch time. She explained that meeting with teachers during lunch is not ideal 
as it could affect the quality of their coaching relationship. “It’s really important to me to establish an easy 
relationship with them, not one where I’m pulling at them and taking away their [break].” 
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Spotlight on 
In-Class Coaching Conferences 


Coaches and teachers described several 
challenges for productive coaching conferences 
that are specific to pre-K, in part because most 
pre-K centers do not have dedicated prep time 
and do not have adequate staff to cover the 
classroom while the coach and teacher meet. We 
asked coaches and teachers to describe how they 
make it work. Spring 2017 interviews with coaches 
and teachers revealed the following strategies: 


Conferencing in the classroom while children are otherwise engaged. Teachers and 
coaches described conferencing while children were napping, having lunch, or had center 
time or free play. However, teachers’ attention was often divided at these times. One coach 
described center time as a good time to sit down and talk with teachers, but added, “It’s 
not as easy as I’m making it [sound] because of supervision. They do have to keep their 
eyes on their children.” 


Coverage. One coach reported that a center director was developing a system to have 
someone in the office cover for a teacher during a conference. Some sites could not 
provide this kind of coverage. One coach noted that, at the beginning of the year, teachers 
were often freed up for conferences, but “it fell apart across the board in most of my 
centers” as the year progressed. Now there is little substitute coverage. The coach said, 
“So I feel like I’m on this treadmill. I have to talk fast, get my points across with the 
post-conference, and schedule my next visit, and make sure the teacher’s feeling okay 
about things.’ Overall, it appeared that few centers had consistent coverage for 
conferences. 


Meeting with two teachers at once. One coach working at a public school site met with 
two pre-K teachers during their common half-hour break. Eventually, later in the year, the 
teachers requested separate meetings. 


Limiting conference time. Some coaches and teachers made do with whatever amount of 
time they could grab. One coach described, “After they implement circle time, then there is 
the post, where | pull them aside. It just takes a few moments to tell them about the pros 
and cons of their presentation and how we can enhance that even more using Blueprint.’ 
In such situations, coaches were able to cover the essentials but may have lacked the 
opportunity to have a full debrief or give the teacher the opportunity to share questions 
and reflections. 
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C. Implementing Blueprint for Early Literacy in the Classroom 


In this section we describe teacher and center director perceptions of the Blueprint materials and 
approaches to instruction, particularly in the early stages when teachers were new to CLI’s approaches. 
We also provide an analysis of teacher implementation of key elements of Blueprint and an in-depth 
examination of the instructional role of assistant teachers. 


Key Findings: Implementing Blueprint in the Classroom 


Upon initially adopting Blueprint in their classrooms, some teachers praised the 
curriculum as well-organized, more complete and engaging, and easier to use than the 
language and literacy components of other curricula. However, others were initially 
resistant to adopting a new approach. 


We learned through spring 2019 surveys that, though almost 80% of surveyed 
teachers felt knowledgeable about all three key elements of Blueprint, less than half of 
teachers reported implementing all three elements of Blueprint in the classroom with 
fidelity, i.e., daily use of Intentional Read Aloud and Message Time Plus and integration 
of Power of Three into the culture of the classroom. 


In response to early findings that some surveyed teachers reported that itis a 
challenge to adapt Blueprint to meet the needs of English Language Learners and 
younger children, CLI developed and shared through coaching some strategies to sup- 
port these efforts. 


Assistant teachers play an important role in instruction in their pre-K classrooms; 
consistent with that, CLI coaches reported that they expect assistant teachers to 
support Blueprint instruction by assisting with MTP and IRA, using Power of Three 
language, taking anecdotal notes during whole-group instruction, and supporting 
Blueprint implementation in small groups. 


1. Adopting Blueprint for Early Literacy 


Respondents from every interview site reported positively about teachers’ responses to 
Blueprint 


When we asked in 2017 interviews for teachers’ initial reactions to Blueprint, often, directors and 
teachers used words like “love” and “excitement” to describe how teachers felt. A director said, “For 

the most part all of them really love it.” Another director said, “I think the teachers understand what’s 
going on, and the more coaching they receive, the more excited they are...they like it.” Another director 
commented on “the passion that the teachers have shown in utilizing it. They took it upon themselves to 
open the box and explore it. That was huge, so I really didn’t have to push it... [Staff] love it. They more 

so want to use Blueprint than our other curriculum.” For one teacher, the presence of Blueprint was an 
incentive to choose her current workplace during the hiring process. “That was a big reason why I wanted 
to work here, was the Blueprint. I just love literacy and having a structured literacy curriculum, and being 
able to show parents, ‘This is what we’re doing.” 
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Some directors and teachers praised Blueprint as well-organized, providing everything 
needed for instruction 


Also in Spring 2017, several directors and teachers said that they found Blueprint easier to implement 
than other curricula they have used in the past or were currently using. Multiple people voiced opinions 
such as, “Everything is right there” or “It’s all there.” A director said, “You don’t have to go out looking for 
material. You don’t have to go out looking for books and resources. Everything is basically right there as a 
blueprint. It’s just a matter of them reading the material and implementing the material. I believe that it’s 
a good curriculum.” A teacher said: 


I just love that it’s a guide. I mean, everything is there for you. It’s supportive for you, so if your 
coach wasn’t there, or you couldn’t get in contact with your coach, you probably wouldn't 
really need her, because you got that guide...They couldn’t have made it easier. Everything is 
aligned, everything is step by step. Do this, that, this, that. Then you go to the back of the book, 
you find out what the activity’s about, what you're doing them for, the purpose, the objective. 


A few teachers also praised the wealth of the more than 100 books available with Blueprint. Interviewees 
described the high-quality selection of books, which also provided multiple ways of addressing a theme. 
One teacher explained: 


I had the same approach [before Blueprint], but we didn’t have as much of a book variety. 

I found myself going to the library, picking out other books. ... If | wanted to get in-depth, it 
was hard. With Blueprint I find that even if I don’t want to stay on theme, there’s another 
something that would match my theme. For instance, we were doing trees, but I also wanted to 
do shrubs and Blueprint had that. 


Teachers and center directors reported a preference for Blueprint relative to other curricula 


In Spring 2017, several interviewees said that, in contrast to other curricula, Blueprint was easier to 
use; offered more engaging and higher-quality materials; and incorporated more skill building. Teachers 
described these aspects of Blueprint: 


e Easier to use: “It’s a little bit more detailed and broken down and a little bit more specific. It 
makes it plain. It makes everything a little bit simpler and easier to understand and to follow, as 
well.” 


More engaging, high quality materials: “There’s something about Blueprint. | think it’s the 
coloring and the attractiveness of the book that keeps [children] looking. When I turn it around, 
it’s like [gasp] so | think it’s the attractiveness of it. It’s short, sweet and to the point.” 


Incorporates more skill building: “[With Creative Curriculum] I didn’t do the letter recognition. 
I didn’t do the punctuation. That part wasn’t there.” 


However, some experienced a learning curve in adopting Blueprint, and some teachers 
were initially resistant to adopting it 


Though many teachers reported that Blueprint was set up in a way that was easy to use, teachers were 
still, according to one coach, “working out all of the newness of the content and coaching” in the first year. 
Therefore, “I’ve noticed the second year is where it takes off.” Teachers referred to needing time to “get 
it,” or just to make the transition from their former way of doing things to the new way with Blueprint. 

A director said, “They’ve gotten used to their daily routine with their circle time, and the calendar, and 
different things that they were doing—they were used to doing, and now trying to switch gears into this 
Message Time Plus, and IRA, and not doing some things that they were used to doing for so long [is a 
challenge].” 
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In addition to a learning curve, multiple factors appeared to contribute to teacher resistance to 
adopting the curriculum, including teachers already feeling overloaded, experienced teachers being 
happy with how they were already teaching literacy, reluctance to change the classroom environment and, 
at a few sites, lack of participation in training. 


Teacher buy-in after resistance was often facilitated by coach or CLI staff support and intervention. 
In another case, a director described working closely with one teacher and encouraging her to fully adopt 
Blueprint. “I’m just telling her it’s not taking anything away from anything that you’re doing because it’s 
wonderful, but there’s always room for improvement.” One coach noted that as she worked with teachers 
and participated in training, they came to see the rationale for and benefits of Blueprint. “For the most 
part they’re enjoying it now.” 


ll. Implementing Key Elements of the Blueprint Curriculum 


CLI recommends that teachers implement Message Time Plus and Intentional Read Aloud every day and 
integrate the Power of Three into the culture of the classroom. 


Though almost 80% of surveyed lead teachers felt knowledgeable about all three key 
elements of Blueprint, less than half were consistently implementing Blueprint with fidelity 


We surveyed lead Blueprint teachers about their knowledge and implementation of the three elements of 
the Blueprint curriculum (Figure 15). 


Figure 15. Key elements of Blueprint for Early Literacy: Percent of lead teachers reporting high levels of 
knowledge and fidelity of implementation, Spring 2019, N=27 
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Source: RFA Teacher Survey, 2018-19 


Note: RFA defines “lead” teacher as the teacher CLI targeted to receive coaching. 


Findings: 


e Nearly all surveyed teachers reported that they were either “very” or “extremely” knowledgeable 
about MTP, IRA, and Power of Three and most (78%) felt knowledgeable about all three elements 
of Blueprint. 
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However, less than half (44%) of surveyed lead teachers reported that they implemented all three 
elements of Blueprint with fidelity in the month prior to the end-of-year survey. 


Examining each key element separately suggests that teachers may need more support in how 
to consistently implement Message Time Plus every day. Nearly all surveyed teachers (93%) 
reported that the felt “very” or “extremely” knowledgeable” about MTP, but only about 60% 
reported daily implementation. Gaps in knowledge and fidelity of implementation were less 
pronounced for Intentional Read Aloud and Power of Three. 


In interviews, some teachers said that they did not implement Blueprint components every 
day because of the need to balance multiple curricula 


One teacher said that she could not do MTP every day because of the need to also implement Creative 
Curriculum and because of the amount of sitting that is developmentally appropriate for children: 


We do [intentional read-alouds] every day. Message time is not done every day only because I 
may have a different activity based off of Creative Curriculum. Message time takes [awhile]. I 
can’t also have them sit on the carpet that long. They're not going to be able to sit there for a 
good 30 minutes. 


One coach noted that other teachers also don’t do IRA and MTP every day. “They say, ‘We just can’t fit it 
in.” Coaches noted that low or inconsistent use of Blueprint can lead to more problems in implementation. 
One coach said, “The teachers doing the Blueprint daily have gotten to that place where they can add on 
more specific writing instruction - shared writing, interactive writing.” This coach went on to say, “The 
ones who aren't doing Blueprint daily feel more harried and scattered and less focused.” In addition, 
teachers who use Blueprint regularly can more easily expand beyond MTP and IRA into other areas such 
as writing, according to CLI staff. 


Adapting Blueprint for younger children and English Language Learners is challenging 


To better understand teacher perceptions of use for all children, we surveyed teachers in spring 2018 
about the adaptability of Blueprint for children with different learning needs (Figure 15). 


Figure 16. Percent of lead teachers reporting ease of adaptability of Blueprint for children with different needs, 
Spring 2018, N=25 teachers 
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Note: Sample sizes vary by item from 24-26 teachers 
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Findings: 


e Some teachers struggle to meet the needs of younger students and English Language Learners. 
Over a third (34%) of teachers reported that it was not easy to adapt Blueprint to meet the needs 
of younger students, and over half (52%) reported similarly for English Language Learners. 


In contrast, only 11% of teachers reported it was not easy to adapt Blueprint for children of 
different skill levels. 


In Spring 2017, some teachers identified challenges differentiating for younger or less-skilled students 
and did not have a shared understanding of whether it was permissible or desirable to adapt Blueprint to 
meet students’ needs. To address this challenge, CLI trained coaches and teachers in strategies to adapt 
Blueprint to meet the different needs of children. One coach described this as learning how to “break 
down a lesson and adapt it to the needs of the classroom.” In Spring 2018 interviews, all four CLI coaches 
reported examples of varied strategies they shared with teachers to help them adapt the curriculum for 
children’s varied needs: 


Creating specific differentiated goals. One coach said, “We tell them to set specific goals for 
children when they’re conferring with each child.” These goals can help teachers plan how to 
support each child’s learning. The coach provided an example: “Maybe if a child is learning the 
beginning sounds of letters, we'll certainly provide...letters with pictures for children who need to 
rely on pictures to understand letters.” 


Using classroom staff strategically. For classrooms with assistant teachers, one coach 
recommended asking the assistant teacher to work with three-year-olds while the lead teacher 
works with older children. 


Adapting seating arrangements. One coach said that they encourage English Language Learners 
or younger students to “sit close to the teacher.” 


Providing small-group work. Coaches encouraged teachers to create a small group of students 
who need more instruction and “reteach” them lessons in a more targeted way. 


Making learning more interactive. Coaches listed interactive activities that could be used 
to engage children in the learning process, including “turn-and-talks,’” songs, and using more 
pictures. 


Checking in on children’s understanding. One coach explained, “When they’re reading a story, 
they should stop at least three times during the course of reading that story and ask open-ended 
questions or questions [relevant] to the story for comprehension to make certain that they do 
understand. You can’t go so fast when you’re working with children with different abilities.” 


Incorporating children’s home languages. A coach described one way of incorporating home 
languages, explaining that if there is a story that is in English and Spanish, “the teacher will read it 
in English and in Spanish and then she will create a follow-up activity that uses the Spanish words 
and English words.” 


In interviews, teachers also reported using Message Time Plus scaffolds, substituting books or spending 
more time on them, shifting lesson objectives, using Blueprint’s tips for differentiating for English 
Language Learners, and connecting with families for extra support. Although differentiation strategies 
varied, teachers had a shared understanding that Blueprint lessons can be adapted to meet the needs of 
the children in their classrooms. 
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Ill. Instructional Role of Assistant Teachers 


CLI targeted some assistant teachers to receive CLI supports in centers with high lead teacher turnover, 
and coaches and lead teachers integrated other assistants into coaching in various, less formalized ways. 
To inform CLI’s expectations for assistant teachers to support Blueprint implementation and CLI’s strategy 
for incorporating assistant teachers into coaching, we investigated the role that assistant teachers play 

in classroom instruction. We asked CLI coaches to describe their expectations for assistants in Blueprint 
implementation in spring 2019 and asked teachers to describe assistants’ roles in classroom instruction 
more generally. 


CLI coaches reported that assistant teachers should play a supporting role in Blueprint 
implementation 


One coach said that assistant teachers must be flexible 

and responsive to the needs of the classroom: “The 

assistant really needs to know where they need to 

be most helpful.” Another coach emphasized that the ; ae : ' 
assistant teacher should be “just as much engaged as the I don’t make a distinction. I don't 
teacher and not just doing paperwork.” A third coach care what degree you have. 
reported that assistant teachers should support Blueprint Children are children, and they 
implementation the same way as the lead teacher. This don’t care either. 
coach said, “I don’t make a distinction. I don’t care what 

degree you have. Children are children, and they don’t 

care either.” 


-CLI coach 


Coaches elaborated on some of the ways that assistants could support Blueprint implementation: 


e Assisting with Message Time Plus and Intentional Read Aloud. Although coaches reported 
that sometimes assistant teachers lead MTP and IRA, one coach said, “for the most part, they 
are acting as assistants to the leads.” Rather than solely being led by the lead teacher, coaches 
recommended that assistant teachers support the implementation of these Blueprint elements. 


Taking anecdotal notes during whole-group instruction. Three coaches reported that they 
expected assistant teachers to support whole-group instruction in one specific way during whole- 
group instruction: taking anecdotal notes. One coach explained that the assistant teacher could 

be “just jotting down who found what at message time, jotting down who answered a question 
about comparing and contrasting characters, really just get some good information for the 
teacher because the teachers in pre-K especially do not have time to do that important work.” This 
coach explained that these notes can inform instruction and support better communication with 
families. Another coach said that if assistant teachers take these notes, “the teachers will know 
where [children are] lacking or how they’re becoming proficient at certain skills.” 


Using Power of Three language. Two coaches said that assistant teachers should be using the 
Power of Three language consistently throughout the day. 


Supporting Blueprint in small groups. Two coaches said that assistant teachers should be 
supporting Blueprint implementation in small groups. One coach explained, “I try to encourage 
the assistants to be aware of the Blueprint applications and understand how to administer them to 
the children and use them with the children.” 
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Consistent with CLI coach expectations, center directors expect assistants to play a 
substantial role in pre-K instruction 


Center directors expect assistant teachers to play a 
role in instruction. All three interviewed directors 
reported that both lead and assistant teachers should 
have a shared understanding of the instructional plans 
and process. Two directors said that lead and assistant 
I’m looking for the same thing teachers should be “co-teaching,” or, as one of these 


from both teachers...I don’t really directors explained, “I’m looking for the same thing 
want to tell that it’s a lead and an from both teachers...I don’t really want to tell that it’s a 


FSS HERE lead and an assistant in there.” The director explained 
: her rationale for this stance, saying, “It could be a day 
-pre-K center director where that lead’s not going to be here, and so I don’t 

want that assistant to be in a classroom acting like an 
assistant.” The third director reported that lead teachers 

should take on more instruction and lesson planning responsibility than assistant teachers. This director 

reported expecting that “the lead teacher is more responsible for the delivery of [Blueprint] instruction, 

and then the assistant...they facilitate sort of the procedures around it.”?” 


Nearly all assistant teachers report that they play a role in whole group instruction 


We surveyed assistant Blueprint teachers to understand the extent to which assistants were leading or 
supporting whole group instruction. In Figure 17, we show teacher responses to their role in whole group 
instruction. 


Figure 17. The role of assistant teachers in whole group instruction, Spring 2019, N=17 assistant teachers 
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Note: We received responses from 17 assistant Blueprint teachers, representing a 65% survey response rate among this group. 


27 Expectations for lead and assistant teachers may be the same in some centers for some aspects of the work, though teachers in these positions typically have different credentials 
and education levels. It is well known that there is a significant wage difference by educational level of early childhood educators; some estimates indicate a $3-4 dollar hourly bump 
for those with a B.A. compared to those with an A.A. (DHHS 2016). 
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Findings: 


e Only 6% of assistant teachers reported that they did not have a role in whole group instruction 
and almost two-thirds (65%) reported that their role in whole group instruction was evenly 
divided between leading and supporting.” 


Consistent with this finding, teachers in all three interviewed sites reported that, in practice, assistant 
teachers shared instructional responsibilities with lead teachers - sometimes evenly. For example, a 

lead and assistant pair at one site reported that they alternated lesson planning and implementing every 
other week. The assistant teacher in another classroom at that site led MTP and IRA, but the lead teacher 
did the lesson planning. Even at a site where the lead teachers reported facilitating most whole group 
instruction, including MTP and IRA, assistants often supported small group instruction. Further, both 
interviewed assistants at that site reported delivering whole group instruction on some days: when the 
lead teacher was out or “it depends on basically how we feel that day.’ One of these assistant teachers said 
that she and her teaching partner were “interchangeable.” Her partner also shared, “We're pretty good at 
picking up where the other one left off.” 


28 The survey response rate for assistant teachers was 65%. Even if all non-responding Blueprint assistants did not play a role in instruction, we would still see over 40% of assistants 
evenly dividing whole group instruction with lead teachers. 
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In this section, we summarize high-level findings from this study and identify implications for CLI’s 
Blueprint for Early Literacy. We end with a discussion of broader implications for early childhood 
decision-makers and practitioners. 


Summary of Findings 


Unlike many early childhood curricula, Blueprint for Early Literacy, coupled with CLI 
professional development supports, demonstrates positive impacts on both children’s 
vocabulary and classroom literacy activities 


Our report documents strong evidence of impact of CLI’s pre-K language and literacy supports, notably 
for children’s vocabulary and classroom literacy activities. These results were achieved in spite of the fact 
that levels of training, coaching, and curriculum implementation suffered because of higher than typical 
year-to-year teacher turnover. We find that children in classrooms implementing Blueprint alongside 
Creative Curriculum showed two-and-a-half months of additional growth in receptive vocabulary relative 
to their peers in business-as-usual classrooms implementing Creative Curriculum only. 


Blueprint is particularly well-suited for supporting the early language and literacy growth 
of children in high-need pre-K environments 


Positive impacts despite high turnover, along with teacher and center director descriptions of the ease of 
use, suggest that Blueprint is particularly effective in high-need pre-K environments. Staffing challenges in 
this study were higher than national averages, with nearly half of lead teachers leaving centers each year. 
This level of turnover posed significant challenges for CLI in providing the level of support they intended 
through this project. In January 2017, CLI targeted teachers in 33 classrooms to receive supports to 
improve outcomes for teachers and children. By Spring 2019, less than half of these teachers remained in 
their positions. As a result, by Spring 2019 fewer than one in five lead teachers in the 11 Blueprint centers 
had been to all three core Blueprint trainings and received 60 hours of coaching. Perhaps because of the 
interruption of supports due to turnover, we found that only 44% of teachers reported implementing all 
three elements of Blueprint—IRA, MTP, and Power of Three—with fidelity. This may be because, as we 
heard in interviews, teachers struggled at times to fit in these components when implementing alongside 
other curriculum. 


Recommendations for CLI 


The evidence of positive impact alongside evidence of inconsistent training and curriculum 
implementation fidelity suggests that Blueprint for Early Literacy is well-suited for implementation in 
challenging pre-K settings. We also interpret these findings to mean that, with more consistent support, 
impacts might be more pronounced. We offer the following suggestions for CLI to consider as a result of 
this study. 


e As CLI continues to support pre-K in high-needs communities, we recommended that CLI offer 
trainings more often so that teachers who are new to centers or who missed trainings for other 
reasons have more opportunities to attend. We also recommend that CLI broaden the scope 
of training and coaching to more intentionally bring in and support assistant teachers, which 
could increase the chances of continuity of practices over time and higher fidelity of Blueprint 
implementation. 


CLI should also consider tailoring its professional development model by center, classroom, and/ 
or teacher in order to maximize impact. Centers, classrooms, and teachers have varied strengths 
and needs; therefore, a one-size-fits-all approach might not be the best match for resources and 
maximum impact. 
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We also suggest CLI consider developing a comprehensive curriculum with content in math, 
science, social studies, and other areas, given evidence that some teachers struggled to implement 
Blueprint for Early literacy with fidelity because of the challenge of implementing alongside other 
classroom curricula. 


Finally, this study did not include an assessment of cost, though generally it can be difficult to 
obtain the resources to implement well-designed in-person coaching models on a broad scale. 

As CLI continues to build out its supports for pre-K, we recommend examining the impact of the 
curriculum with and without coaching and exploring an analysis of cost-benefit of various models 
of service delivery in order to guide programs looking to invest in a new curriculum. 


Implications for early childhood educators and policy makers 


Because 4- and 5-year-old children with higher levels of early vocabulary and early literacy skills 
consistently have greater levels of academic success in elementary and middle school, state- and locally 
funded pre-K programs across the country have expanded significantly.” Evaluations of these programs 
have confirmed that enrolled children have better outcomes.*° However, for many preschool programs, 
achieving instructional quality is a struggle.*! Interventions that appear to work in demonstration trials 
fail to maintain positive impacts when programs are scaled. 


This study of the impact of a curriculum supplement to improve instructional quality and child outcomes 
was met with many of the challenges that programs supporting pre-K teachers face when taken to scale, 
including high teacher turnover and difficulties implementing in-person coaching associated with staffing 
coverage. As such, this study offers lessons for early childhood educators and state and local policy 
makers, in Pennsylvania and more broadly. 


The benefits of programs like CLI’s may be significant but are unlikely to be large enough 
to address gaps driven by poverty. The level of need for supporting the school readiness 

of preschool children in Philadelphia, which is shared by other large cities with a history of 
underfunded educational systems and intergenerational poverty, is significant. When we assessed 
children’s vocabulary in the fall, prior to their pre-K year, the children in this study were over a 
year behind national average in vocabulary development. While Blueprint training, coaching, and 
curriculum implementation accelerated growth by over two months, the benefits do not come 
close to addressing developmental gaps that, as research shows, will grow as children enter 
formal schooling.*? 


Providing professional development in the context of high turnover is an unavoidable 
implementation challenge that requires specific strategies to overcome. Pre-K staff turnover 
in Philadelphia, and nationally, is an unavoidable implementation challenge, given current teacher 
wages. Pre-K curriculum developers must contend with this challenge and create materials that 
are, at minimum, easy to implement without significant training. Our analysis of CLI also suggests 
that developers should consider offering flexible training options, potentially online training videos 
and/or artifacts that could support directors, for onboarding teaching staff mid-year 

so they can get the most out of their instructional materials. 


29 Friedman-Krauss, A. H., Barnett, W. S., Garver, K. A., Hodges, K. S., Weisenfeld, G.G., & Dicrecchio, N. (2019). The State of Preschool 2018. Retrieved from: 
http://nieer.org/wp-content/uploads/2019/08/YB2018_Full-ReportR3wAppendices.pdf 

30 Wong, V.C., Cook, T. D., Barnett, W. S., & Jung, K. (2007). An effectiveness-based evaluation of five state pre-kindergarten programs. Journal of Policy Analysis and Management, 
27, 122-154. https://doi.org/10.1002/pam.20310 

31 Burchinal, M., Kainz, K. & Cai, Y. (2011). How well do our measures of quality predict child outcomes? A meta-analysis and coordinated analysis of data from large-scale studies of 
early childhood settings. \n M. Zaslow, |. Martinez-Beck, K. Tout, & T Halle (Eds), Quality Measurement in Early Childhood Settings (pp 11-31). Baltimore, MD: Brookes. 

32 Duncan and Magnuson 2011 The nature and impact of early achievement skills, attention skills, and behavior problems. In Duncan and Murnane (eds) Whither Opportunity? 
Rising Inequality, Schools, and Children’s life chances (p 47-69). New York, NY: Russell Sage Foundation. 


Boosting Children’s Language and Literacy 
RESEARCH for ACTION Skills through Blueprint 


Paraprofessionals in pre-K play an important instructional role in the classroom and 
expectations for their role should be considered when designing and implementing 
supports, with more than half of those we surveyed reporting leading whole group instruction 
at least some of the time. In talking with CLI coaches and center directors, we found that 
expectations for assistant teachers are high; in our interviews, we found that respondents wanted 
to “not be able to tell the difference” between the lead and assistant teacher when observing 
instruction. However, many programs do not intentionally train or coach paraprofessionals in 
the classroom. Though CLI invited assistant teachers to trainings, and some assistants received 
coaching because they were the more stable staff in the classroom over time, we urge those who 
make decisions about who and how to train to recognize the expectations for assistant teachers 
and give them the tools to support implementation of curricular activities alongside the teacher 
formally leading instruction. 


Boosting Children’s Language and Literacy 
RESEARCH for ACTION Skills through Blueprint 


Appendix: Data Collection and Analytic Methods 


A. Teacher Survey Sample Description 


Instrument development. RFA developed a questionnaire with 30 open- and closed-response 
items for teachers to generate cross-sectional data on teacher perceptions of their knowledge and 
ability to implement effective practices in early literacy instruction and student learning and 
engagement. We also asked Blueprint teachers an additional 30 questions to reflect on experiences 
with Blueprint professional development and implementation of the Blueprint curriculum. 


Sampling frame and survey administration. In May 2019, RFA used Qualtrics to program and 
administer a web-based survey to all lead teachers who were working at study sites who had 
consented to our study (N=78). The survey was live for four weeks, and three reminder emails were 
sent to non-respondents during that time. Teachers provided their email addresses when they 
signed consent forms. 


Response rate. Of the 74 lead teachers that consented to the study, RFA received surveys from 55, a 
response rate of 74%. The lead teacher response rate was not statistically different across 
treatment and comparison sites (75% vs 73%, respectively, p=0.341). We also extended an 
invitation for assistant teachers in Blueprint classrooms to complete the survey (N=26). In 
treatment classrooms, we received 17 surveys from assistants, a response rate of 65%. 


A note on representativeness of lead teacher survey data. Our analysis does not account for 
variation in the number of teachers per site and the number of teachers responding per site. This is 
due to asmall overall sampling size. That is, the data cannot support strategies to account for the 
structure of the data, e.g., weighting results based on the number of classrooms per center or 
estimating multi-level models. 


B. Interviews 


Over the course of the study, we conducted 69 interviews with CLI staff, coaches, center directors, 
and lead and assistant teachers. Our research focus shifted each year to allow flexibility in our data 
collection to dig deeper into implementation successes and challenges we saw each year. 


Protocol development. RFA developed open-ended and semi-structured interview protocols to 
understand each year of study’s respective focus. 


Focus by year. The focus of interviews shifted from year to year, to allow for in-depth investigation 
into specific strategies, successes, and challenges as the study progressed. 


e In Year One, the focus of interviews was to understand general strengths, challenges, 
and perceived impact of the Blueprint intervention. 
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e In Year Two, the focus was to deepen understanding of implementation, especially 
challenges that were raised in Year One, and perceived impacts of CLI training and 
coaching on teacher and student outcomes. 


e In Year Three, the focus was to deepen understanding of the implementation of 
Blueprint coaching and the role of assistant teachers in supporting a Blueprint 
classroom. 


Participants. Each year, RFA interviewed relevant CLI staff (n=4 in Year One, n=3 in Year Two, n=2 
in Year Three) and all four CLI coaches. We also interviewed treatment center directors and 
teachers, strategically sampled for deep understanding of each year’s respective focus. 


Year One. We interviewed all nine directors of centers for which we had approval to conduct 
research at the time. We conducted interviews with a total of 12 lead teachers at seven of the nine 
treatment centers where we received research approval. We chose these seven sites to achieve 
variation across several indicators, including type of center; geographic location; size; and literacy 
curricula used. Our goal was to interview two teachers at each center, and we used the following 
criteria to identify interviewees if there were more than two pre-K teachers: 


e In 9 of 11 treatment sites, one teacher began receiving CLI supports in Fall 2016, as part 
of another grant. At sites which began implementation in one classroom in Fall 2016, 
we interviewed the early implementation teacher and a second teacher. At half of the 
sites, we asked to interview a teacher who was struggling with Blueprint or whose 
engagement the director was unsure of; at the other half we asked to interview a 
teacher very engaged in Blueprint. 


e Atsites without a fall implementation classroom, we asked the director to identify one 
teacher very engaged in Blueprint and one teacher who was struggling or whose 
engagement the director was unsure of. 


At one site, we learned during our interview that the teacher began implementing Blueprint prior 
to school year 2016-17, and we excluded this interview from our analysis. 


Year Two. RFA interviewed four directors and eight lead teachers in four treatment centers. We 
strategically sampled directors and teachers to go deeper into issues that were raised in Year One: 


e Director support and knowledge of Blueprint. CLI staff reported that two selected 
sites had high director support and knowledge of Blueprint and two selected sites had 
low director support and knowledge of Blueprint. 


e Teacher turnover. Two sites had high teacher turnover; two sites had low teacher 
turnover. 


e Differentiation of Blueprint. Teachers in our sample had both high and low 
proportions of English Language Learners and younger learners. 
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e Implementing Blueprint with Creative Curriculum. All four sites balanced using 
Blueprint with Creative Curriculum. 


e Experience with Blueprint. Half of the interviewed teachers were new to 
implementing Blueprint in Year Two; the other half implemented Blueprint in Year One 


e Strength of implementation. Half of the interviewed teachers were identified by CLI 
coaches as strong Blueprint implementers; the other half were identified as weaker 
Blueprint implementers. 


Year Three. RFA interviewed three directors, six lead teachers, and six assistant teachers in three 
treatment centers. We strategically sampled treatment centers and teachers to maximize variation 
in teacher tenure. In one center, assistant teachers taught at the center for longer than lead 
teachers, in the second, both lead teachers taught at the center throughout the study but had new 
assistant teachers this year. In the last center, all teachers taught at the center throughout the entire 
study. These sites also varied by type and included a pre-K center based in a school district school, 
an independent early childhood center, and an early childhood center that is part of a provider 
network. Each of the sampled centers had a different CLI coach. 


Data analysis. Interviews were recorded (with consent), transcribed, and coded using the dedoose 


web application. The data were then systematically analyzed using a multi-stage analytic memo 
process. 


C. Assessing Baseline Equivalence for Impact Analysis 


Comparison Site Selection 


In Year One of RFA’s CLI Blueprint evaluation, CLI and RFA worked together to recruit a total of 26 
STAR 3 and 4 pre-K centers in North, West, and South Philadelphia. Eleven of the centers received 
Blueprint training and coaching January 2017-June 2018, and 15 served as a comparison group. 
Drawing from STAR 3 and 4 centers across the city, we already reduced much of the variation in 
pre-K center quality. Additional key aspects of comparison site selection included: 


e High need. To create a matched comparison group, CLI also selected comparison sites 
from areas with a high need for quality pre-K. 


e Geography. CLI selected sites in the same geographic areas as the treatment sites. 
e Approval by district. Because several SDP sites participated in the treatment group 
and because SDP provides supports to many centers, CLI submitted the list to SDP for 


their sign-off. 


e Relationship with CLI. CLI sought sites with no previous relationship with CLI. 
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Methodological Strategies to Address Non-random Selection of Centers 


In Year Two, we employed several strategies to address the non-random selection of centers in 
order to estimate impacts of Blueprint on teacher and student outcomes. 


Specific strategies included: 


e Collecting data via teacher surveys on characteristics of centers, directors, and teachers 
to more fully assess differences in the contexts of treatment and comparison centers; 


e Collecting pre-test student outcome data to account for pre-existing differences across 
treatment groups; and 


e Drawing from an over-recruited sample of comparison centers to identify a subset that 
most closely matched the centers in our treatment group. 


We recommended dropping two centers for non-statistical reasons. One center served a high 
proportion of students with special needs, and the other had consistently had poor response rates 
across data collection activities. 


To identify an additional two centers to exclude, we estimated a statistical model predicting 
treatment status based on two variables that are highly correlated with other center 
characteristics: baseline PPVT scores and whether the center was a school district or community 
provider site. Estimated predicted treatment status across treatment and comparison centers 
revealed limited overlap at the high end of the distribution of propensity scores. This created a 
disproportionately lower average propensity for treatment among comparison centers. To address 
this, we excluded the two comparison sites with the lowest propensity for treatment. This resulted 
in a more even distribution at the lower end to reduce disproportionately low propensity 
comparison sites. 


Baseline Equivalence of Treatment and Control Centers 


As Tables 1A and 1B show, the treatment and comparison sites were statistically equivalent on key 
characteristics relevant to early literacy. Importantly, the pre-test measure of the main outcome of 
interest is evenly distributed across groups. We examined baseline equivalence for child outcomes 
each year of the impact study. 
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Table 1A. Statistical comparison of center characteristics, t-tests, Year Two 


Business-as- 
Blueprint UESTE) 

Centers Centers 
Center Characteristics (ee aD) (ee aD) 
Number of school district sites 2 2 - - 
Number of pre-K classes 3.09 (2.12) 3.54 (1.63) 0.45 0.579 
Average class size 15.88 (4.06) 15.98 (3.06) 0.10 0.951 
Average prop ELL per class 0.15 (0.29) 0.13 (0.24) -0.01 0.912 
Average prop IEP per class 0.05 (0.05) 0.08 (0.05) 0.03 0.2114 
pvetac ei on sunuen se” pel 0.10 (0.08) 0.10 (0.04) <0.01 0.958 
class 
Prop classes - Head Start 0.33 (0.40) 0.42 (0.35) 0.09 0.563 
Prop classes - PreK Counts 0.23 (0.40) 0.39 (0.38) 0.15 0.367 
Standard PPVT score 93.12 (7.00) 91.37 (3.93) -1.75 0.479 


Table 1B. Statistical comparison of center characteristics, t-tests, Year Three 


Business-as- 
Blueprint UST EE) 

Centers Centers 
Center Characteristics (ee aD) (ee aD) 
Number of school district sites 2 2 - - 
Number of pre-K classes 4.65 (2.46) 4.30 (1.73) -0.35 0.480 
Average class size 14.21 (4.96) 14.70 (3.96) 0.49 0.635 
Average prop ELL per class 0.10 (0.24) 0.21 (0.31) 0.11 0.1014 
Average prop IEP per class 0.11 (0.10) 0.15 (0.11) 0.04 0.1114 
ee eee ee Owner 0.10 (0.16) 0.11 (0.13) 0.02 0.611 
4-years old in May per class 
Prop classes - Head Start 0.41 (0.50) 0.43 (0.50) 0.01 0.910 
Prop classes - PreK Counts 0.21 (0.41) 0.28 (0.45) 0.07 0.497 
Average Standard PPVT score 94.14 (14.61) 93.29 (12.12) -0.85 0.331 
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D. Observations of Classroom Language and Literacy 
Environments 


The Early Literacy and Language Classroom Observation Research Edition Pre-K Tool (ELLCO) 
measures teacher language and literacy instruction and general classroom environments. Each 
ELLCO item is rated with a five-point anchored scale that provides descriptions of ratings at each 
level. 


ELLCO Data Constructs. The 19 ELLCO observation items are grouped into five constructs:1 


e The Classroom Structure construct measures the physical environment of the 
classroom, including the layout, traffic flow, an inventory of the materials, as well as 
how the children use them. It also includes observations of classroom management and 
adult roles within the classroom. 


e The Curriculum construct measures the use of time throughout the observation, the 
integration of themes, and the recognition of child diversity and choice in the daily 
routine. 


e The Language Environment construct measures the discourse climate, how teachers 
interact with the children, and how they build vocabulary and phonological awareness 
through extended conversations. 


e The Books construct measures the organization, characteristics, availability, and use of 
books for learning. It also measures the teacher’s approach to reading a book aloud to 
children and their general engagement during the read aloud. 


e Finally, the Print and Early Writing construct measures the use of environmental print, 
writing materials, and writing instruction. 


These five constructs are further grouped into two subscales — the General Classroom 
Environment subscale and the Language and Literacy subscale. 


e The General Classroom Environment subscale is a composite of the Classroom Structure 
construct and the Curriculum construct. 


e The Language and Literacy subscale is a composite of the Language Environment, Books 
and Book Reading, and Print and Early Writing constructs. 


Smith et al. (2008) report an internal consistency of 0.86 for the Language and Literacy subscale 
and of 0.83 for the General Classroom Environment subscale. ELLCO observational scores are 


1 Smith, M. W., Brady, J. P., & Anastasopoulos, L. (2008). User's Guide to the Early Language & Literacy Classroom Observation Pre-K Tool. 
Baltimore: Paul H. Brookes Publishing Co. 
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predictive of children’s receptive vocabulary (as measured by the Peabody Picture Vocabulary 
Test) and early literacy scores (as measured by the Profile of Early Literacy Development). 


Observation training and inter-rater reliability. To establish inter-rater reliability, RFA 
observers received ELLCO training and practiced observing classrooms in March-May 2017. 
Practice sessions and debriefing conversations were used to calibrate ratings after each 
observation to achieve inter-rater reliability (i.e., agreement within one point on each item). 


E. Student Outcomes: Direct Assessment 


In Years Two and Three, RFA collected student vocabulary assessment data by administering the 
PPVT to students enrolled in study centers. Our sample included children who were proficient in 
English? and whose caretakers did not opt them out of participating. RFA staff and external 
consultants, hired by RFA, collected Year Two baseline data from October-December 2017, Year 
Two post data from March-June 2018, Year Three baseline data from October-December 2017, and 
Year Three post data from March-June 2018. Assessment data were considered valid if students 
passed the PPVT training items, had a raw score of at least 4, and completed the assessment.3 


e Final Year Two sample. Students for whom RFA had valid data from fall and spring 
were included in the analysis (N=808, 70% of the full sample). There were not 
differential response rates across treatment and comparison groups (70% vs 69%). The 
final sample includes data from 32/34 treatment classrooms and 39/40 comparison 
classrooms. 


o Fall 2017 sample. Of the 1,242 students enrolled in study centers in Fall 2017, 
RFA collected valid assessment data from 1,092 students, a response rate of 
88%. The sample was comprised of 458 students from treatment centers and 
634 students from comparison centers. 


o Spring 2018 sample. Of the 1,121 students enrolled in study centers in Spring 
2018, RFA collected valid assessment data from 1,063 students, a response rate 
of 95%, In the spring, 487 students were from treatment centers and 576 
students were from comparison centers. 


e Final Year Three sample. Children for whom RFA had valid data from fall and spring 
were included in the analysis (N=783, 88.2% of the full sample). There were not 
differential response rates across treatment and comparison groups (89% vs 87%). The 
final sample includes data from 33/36 treatment classrooms and 38/38 comparison 
classrooms. 


2 We initially collected baseline data for students proficient in Spanish using the Spanish-language version of the PPVT, the TVIP. We 
discontinued use of the TVIP during spring data collection because we determined the Spanish-language version was not comparable to 
the English version based on the following reasons: the TVIP is in black and white and less engaging for young children; the scoring 
process is different for the TVIP; the TVIP does not include the Growth Scale Value (GSV); and there are fewer total items in the TVIP, 
providing an advantage to earn a higher score to students taking the PPVT. In practice, we did not find these assessments to be parallel 
and ultimately excluded TVIP data from the sample. 


3 Dunn, L. M., Dunn, D. M., Lenhard, A., Lenhard, W., & Suggate, S. (2015). Peabody Picture Vocabulary Test [manual]. Pearson. 
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o Fall 2018 sample. Of the 1,010 children enrolled in study centers in Fall 2018, 
RFA collected valid assessment data from 945 children, a response rate of 
93.5%. The sample was comprised of 426 children from treatment centers and 
519 children from comparison centers. 


o Spring 2019 sample. Of the 1,136 children enrolled in study centers in Spring 
2019, RFA collected valid assessment data from 1,050 children, a response rate 
of 92.4%, In the spring, 500 children were from treatment centers and 550 
children were from comparison centers. 


Dependent variable. The student outcomes measure was the Growth Scale Value (GSV) score from 
the PPVT assessment. The GSV was used because it can measure change in scores over time ona 
single, continuous scale, allowing us to compare scores over time and to determine a program’s 
effectiveness.* 


Multi-level model. RFA estimated a multi-level random intercept model, adjusting for the 
following student-level characteristics: baseline PPVT growth scores, ELL status, and age. Multi- 
level modeling provided an estimate of the difference in average GSV scores between treatment and 
comparison groups, accounting for clustering of students within centers and student-level controls. 


PPVTSpringi; = Yoo + YoiTreatment + ¥i9PPVT Fall; + y2oELLStatus; + y3oAge; + ei; + Uo; 


Student-level model (Level 1). We began by modeling a child’s GSV from spring PPVT as a 
function of select student-level characteristics (i.e., baseline scores, child’s age, ELL status) anda 
student-level random error (i.e., e; ): 


PPVTSpring;; = Boj + 1 ;PPVT Fall; + B2;ELLStatus; + B3;Age; + e 


Where, fo; is the mean GSV score of center j, accounting for differences in children’ baseline scores, 


age, and ELL status. e; is the difference between an individual child’s PPVT score and their center’s 
mean. 


Center-level model (Level 2). At the Level 2, we specified the adjusted center-level means are a 
function of the treatment condition. However, slope coefficients 6,, 62, and 63 are assumed to be 
fixed at level 2. We examined an alternative specification that allowed these slope coefficients to 
vary with a random error at the center-level. This alternative model was rejected because it did not 
improve overall model fit compared to the model presented here. 


Boj =Yoo + Yo1Treatment + Uo; 


Pr; = Y10 
B2; = Y20 
B3; = Y30 


4 Ibid. 
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Where Yoo represents the grand mean of comparison centers, where treatment = 0 (i-e., comparison 
centers); Yo; represents the treatment effect, which is the difference between the treatment mean 
and comparison mean, adjusting for student-level covariates; y19, Y29, and Y39 represent the effects 


of PPVTFall, ELLStatus, and Age on the dependent variable (PPVTSpring), respectively; and uo ; 
represents the center-level error term, i.e., the difference between a center’s mean and the grand 


mean of all children. 


Table 2A. Mixed-model coefficients for Year Two, N=808 


Covariates Coefficient Standard error | P-value 

Treatment effects (center level) 3.60** 1.30 0.005 
Baseline (PPVT Fall GSV, student level) 0.62*** 0.02 0.000 
ELL status (student level) -1.76 1.64 0.001 
Age, under 4 in May (student level) -4,46*** 1.36 0.284 
Intercept (adjusted grand mean) 49.35*** 2.63 0.000 


*p<0.05; **p<0.01; ***p<0.001 
Table 2B. Mixed-model coefficients for Year Three, N=783 


Covariates Coefficient Standard error | P-value 
Treatment effects (center level) 2.65* 1.15 0.021 
Baseline (PPVT Fall GSV, student level) 0.71*** 0.02 <0.001 
ELL status (student level) 21 1.45 0.886 
Age, under 4 in May (student level) -4.03* ** 1.17 0.001 
Intercept (adjusted grand mean) 37.53*** 2.64 <0.001 

*p<0.05; **p<0.01; ***p<0.004 

Table 2C. Mixed-model coefficients for combined child sample, N=1,591 
Covariates Coefficient Standard error | P-value 
Treatment effects (center level) 3.05*** 0.922 0.001 
Baseline (PPVT Fall GSV, student level) 0.66*** 0.02 <0.001 
ELL status (student level) -0.63 1.12 0.571 
Age, under 4 in May (student level) -4,33%*** 0.90 <0.001 
Study year (Year Three==1) -1.58** 0.51 0.002 
Intercept (adjusted grand mean) 37.53% ** 2.64 <0.001 


*p<0.05; **p<0.01; ***p<0.001 


Effect size. RFA calculated effect size as the difference between treatment and comparison groups’ 
mean GSV scores divided by the standard deviation of the normed population. The standard 
deviation of the GSV of the normed population sample varies by age. RFA calculated the effect size 
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of Blueprint based on the standard deviation of the normed population for the age group at the time 
of spring assessments (weighted average: 15.25). 


Months of growth. RFA calculated average the impact of CLI Blueprint on study vocabulary in 
terms of the additional months of growth that children in treatment centers gained relative to their 
peers in control centers. To do this, we used normed scores across the age groups of the children in 
our sample, which is equivalent to 1.28 points of growth in PPVT scores per month. On average 
treatment children scored 3.05 points higher than their peers, translating to 2.38 months of 
additional growth (3.05/1.28=2.38). 
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